agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v19 6/8] Row pattern recognition patch (docs). 390+ messages / 2 participants [nested] [flat]
* [PATCH v19 6/8] Row pattern recognition patch (docs). @ 2024-05-14 23:26 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Tatsuo Ishii @ 2024-05-14 23:26 UTC (permalink / raw) --- doc/src/sgml/advanced.sgml | 82 ++++++++++++++++++++++++++++++++++++ doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++ doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++- 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 755c9f1485..b0b1d1c51e 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -537,6 +537,88 @@ WHERE pos < 3; <literal>rank</literal> less than 3. </para> + <para> + Row pattern common syntax can be used to perform row pattern recognition + in a query. The row pattern common syntax includes two sub + clauses: <literal>DEFINE</literal> + and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines + definition variables along with an expression. The expression must be a + logical expression, which means it must + return <literal>TRUE</literal>, <literal>FALSE</literal> + or <literal>NULL</literal>. The expression may comprise column references + and functions. Window functions, aggregate functions and subqueries are + not allowed. An example of <literal>DEFINE</literal> is as follows. + +<programlisting> +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +</programlisting> + + Note that <function>PREV</function> returns the price column in the + previous row if it's called in a context of row pattern recognition. Thus in + the second line the definition variable "UP" is <literal>TRUE</literal> + when the price column in the current row is greater than the price column + in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when + the price column in the current row is lower than the price column in the + previous row. + </para> + <para> + Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be + used. <literal>PATTERN</literal> defines a sequence of rows that satisfies + certain conditions. For example following <literal>PATTERN</literal> + defines that a row starts with the condition "LOWPRICE", then one or more + rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that + "+" means one or more matches. Also you can use "*", which means zero or + more matches. If a sequence of rows which satisfies the PATTERN is found, + in the starting row of the sequence of rows all window functions and + aggregates are shown in the target list. Note that aggregations only look + into the matched rows, rather than whole frame. On the second or + subsequent rows all window functions are NULL. Aggregates are NULL or 0 + (count case) depending on its aggregation definition. For rows that do not + match on the PATTERN, all window functions and aggregates are shown AS + NULL too, except count showing 0. This is because the rows do not match, + thus they are in an empty frame. Example of a <literal>SELECT</literal> + using the <literal>DEFINE</literal> and <literal>PATTERN</literal> clause + is as follows. + +<programlisting> +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); +</programlisting> +<screen> + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | 0 + company1 | 2023-07-03 | 150 | | | 0 + company1 | 2023-07-04 | 140 | | | 0 + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | 0 + company1 | 2023-07-08 | 130 | | | 0 + company1 | 2023-07-09 | 120 | | | 0 + company1 | 2023-07-10 | 130 | | | 0 +(10 rows) +</screen> + </para> + <para> When a query involves multiple window functions, it is possible to write out each one with a separate <literal>OVER</literal> clause, but this is diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 17c44bc338..8dbab31300 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -23124,6 +23124,7 @@ SELECT count(*) FROM sometable; returns <literal>NULL</literal> if there is no such row. </para></entry> </row> + </tbody> </tgroup> </table> @@ -23163,6 +23164,59 @@ SELECT count(*) FROM sometable; Other frame specifications can be used to obtain other effects. </para> + <para> + Row pattern recognition navigation functions are listed in + <xref linkend="functions-rpr-navigation-table"/>. These functions + can be used to describe DEFINE clause of Row pattern recognition. + </para> + + <table id="functions-rpr-navigation-table"> + <title>Row Pattern Navigation Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>prev</primary> + </indexterm> + <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the previous row; + returns NULL if there is no previous row in the window frame. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>next</primary> + </indexterm> + <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the next row; + returns NULL if there is no next row in the window frame. + </para></entry> + </row> + + </tbody> + </tgroup> + </table> + <note> <para> The SQL standard defines a <literal>RESPECT NULLS</literal> or diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index 066aed44e6..8f18718d58 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl The <replaceable class="parameter">frame_clause</replaceable> can be one of <synopsis> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] </synopsis> where <replaceable>frame_start</replaceable> @@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. </para> + <para> + The + optional <replaceable class="parameter">row_pattern_common_syntax</replaceable> + defines the <firstterm>row pattern recognition condition</firstterm> for + this + window. <replaceable class="parameter">row_pattern_common_syntax</replaceable> + includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST + ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls + how to proceed to next row position after a match + found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the + default) next row position is next to the last row of previous match. On + the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next + row position is always next to the last row of previous + match. <literal>DEFINE</literal> defines definition variables along with a + boolean expression. <literal>PATTERN</literal> defines a sequence of rows + that satisfies certain conditions using variables defined + in <literal>DEFINE</literal> clause. If the variable is not defined in + the <literal>DEFINE</literal> clause, it is implicitly assumed + following is defined in the <literal>DEFINE</literal> clause. + +<synopsis> +<literal>variable_name</literal> AS TRUE +</synopsis> + + Note that the maximu number of variables defined + in <literal>DEFINE</literal> clause is 26. + +<synopsis> +[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ] +PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...] +DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...] +</synopsis> + </para> + <para> The purpose of a <literal>WINDOW</literal> clause is to specify the behavior of <firstterm>window functions</firstterm> appearing in the query's -- 2.25.1 ----Next_Part(Wed_May_15_09_02_03_2024_008)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v19-0007-Row-pattern-recognition-patch-tests.patch" ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-06-16 11:54 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-06-16 11:54 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 2268cc277bc..127d4415084 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -609,11 +608,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -627,11 +623,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -705,6 +696,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -881,11 +874,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -901,8 +892,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -922,6 +912,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -932,33 +924,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2346,21 +2341,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2372,6 +2391,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2380,76 +2402,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v01-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
* [PATCH 1/8] Use tuple slot to pass tuples for rewriting. @ 2026-07-15 08:37 Antonin Houska <[email protected]> 0 siblings, 0 replies; 390+ messages in thread From: Antonin Houska @ 2026-07-15 08:37 UTC (permalink / raw) This patch tries to adopt the preferable way of handling tuples, i.e. pass the containing tuple slot rather than the actual tuple. The motivation is that heap_insert_for_repack() will need a slot in the near future, in order to call ExecInsertIndexTuples(). Also, in order to set values of dropped attributes to to NULL, it seems more compact to pass a tuple slot as a workspace than two arrays (one for values and one for nulls). --- src/backend/access/heap/heapam_handler.c | 199 +++++++++++++---------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bf87430cf01..4db8a16f691 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -48,14 +48,13 @@ #include "utils/rel.h" #include "utils/tuplesort.h" -static void reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate); -static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull, - BulkInsertState bistate); -static HeapTuple reform_tuple(HeapTuple tuple, Relation OldHeap, - Relation NewHeap, Datum *values, bool *isnull); +static void reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate); +static void heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, + BulkInsertStateData *bistate); +static bool tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc); +static void clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform); static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, HeapTuple tuple, @@ -603,11 +602,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, bool is_system_catalog; Tuplesortstate *tuplesort; TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); TupleTableSlot *slot; - int natts; - Datum *values; - bool *isnull; + TupleTableSlot *reform_slot; BufferHeapTupleTableSlot *hslot; BlockNumber prev_cblock = InvalidBlockNumber; bool concurrent = snapshot != NULL; @@ -621,11 +617,6 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, */ Assert(RelationGetTargetBlock(NewHeap) == InvalidBlockNumber); - /* Preallocate values/isnull arrays */ - natts = newTupDesc->natts; - values = palloc_array(Datum, natts); - isnull = palloc_array(bool, natts); - /* * In non-concurrent mode, initialize the rewrite operation. This is not * needed in concurrent mode. @@ -699,6 +690,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, slot = table_slot_create(OldHeap, NULL); hslot = (BufferHeapTupleTableSlot *) slot; + reform_slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsVirtual); /* * Scan through the OldHeap, either in OldIndex order or sequentially; @@ -875,11 +868,9 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, int64 ct_val[2]; if (!concurrent) - reform_and_rewrite_tuple(tuple, OldHeap, NewHeap, - values, isnull, rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* * In indexscan mode and also VACUUM FULL, report increase in @@ -895,8 +886,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, index_endscan(indexScan); if (tableScan != NULL) table_endscan(tableScan); - if (slot) - ExecDropSingleTupleTableSlot(slot); + ExecDropSingleTupleTableSlot(slot); /* * In scan-and-sort mode, complete the sort, then read out all live tuples @@ -916,6 +906,8 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_WRITE_NEW_HEAP); + slot = MakeSingleTupleTableSlot(RelationGetDescr(OldHeap), + &TTSOpsHeapTuple); for (;;) { HeapTuple tuple; @@ -926,33 +918,36 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap, if (tuple == NULL) break; + /* + * XXX Ideally we should use tuplesort_gettupleslot() above, but + * it retrieves minimal tuples and tuplesort_puttupleslot() cannot + * get the tuple descriptor from tuplesort created by + * tuplesort_begin_cluster(). + */ + ExecStoreHeapTuple(tuple, slot, false); + n_tuples += 1; if (!concurrent) - reform_and_rewrite_tuple(tuple, - OldHeap, NewHeap, - values, isnull, - rwstate); + reform_and_rewrite_tuple(slot, reform_slot, rwstate); else - heap_insert_for_repack(tuple, OldHeap, NewHeap, - values, isnull, bistate); + heap_insert_for_repack(NewHeap, slot, reform_slot, bistate); /* Report n_tuples */ pgstat_progress_update_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, n_tuples); } + ExecDropSingleTupleTableSlot(slot); tuplesort_end(tuplesort); } + ExecDropSingleTupleTableSlot(reform_slot); + /* Write out any remaining tuples, and fsync if needed */ if (rwstate) end_heap_rewrite(rwstate); if (bistate) FreeBulkInsertState(bistate); - - /* Clean up */ - pfree(values); - pfree(isnull); } /* @@ -2340,21 +2335,45 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate, * currently only known to happen as an after-effect of ALTER TABLE * SET WITHOUT OIDS. * - * So, we must reconstruct the tuple from component Datums. + * So, we must reconstruct the tuple from component Datums. 'reform' slot + * is a workspace for this reconstruction. */ static void -reform_and_rewrite_tuple(HeapTuple tuple, - Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, RewriteState rwstate) +reform_and_rewrite_tuple(TupleTableSlot *src, TupleTableSlot *reform, + RewriteState rwstate) { - HeapTuple newtuple; + HeapTuple tuple, + newtuple; + bool shouldFree, + shouldFreeNew; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + /* + * The old tuple will not be modified, so do not request materialization. + * (A copy can be created for specific slot type though, e.g. + * TTSOpsMinimalTuple.) + */ + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + + /* No need to materialize, copy will be created anyway. */ + Assert(TTS_IS_VIRTUAL(reform)); + newtuple = ExecFetchSlotHeapTuple(reform, false, &shouldFreeNew); + } + else + { + newtuple = heap_copytuple(tuple); + shouldFreeNew = true; + } /* The heap rewrite module does the rest */ rewrite_heap_tuple(rwstate, tuple, newtuple); - heap_freetuple(newtuple); + if (shouldFree) + heap_freetuple(tuple); + if (shouldFreeNew) + heap_freetuple(newtuple); } /* @@ -2366,6 +2385,9 @@ reform_and_rewrite_tuple(HeapTuple tuple, * information). Thus we must use heap_insert() both during the * catch-up and here. * + * 'reform' is a slot to use for tuple "reforming", typically to get set + * values of dropped columns to NULL. + * * We pass the NO_LOGICAL flag to heap_insert() in order to skip logical * decoding: as soon as REPACK CONCURRENTLY swaps the relation files, it drops * this relation, so no logical replication subscription should need the data. @@ -2374,76 +2396,75 @@ reform_and_rewrite_tuple(HeapTuple tuple, * case. */ static void -heap_insert_for_repack(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull, BulkInsertState bistate) +heap_insert_for_repack(Relation rel, TupleTableSlot *src, + TupleTableSlot *reform, BulkInsertStateData *bistate) { - HeapTuple newtuple; + HeapTuple tuple; + bool shouldFree; + TupleTableSlot *slot; - newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull); + tuple = ExecFetchSlotHeapTuple(src, false, &shouldFree); + if (tuple_needs_reform(tuple, src->tts_tupleDescriptor)) + { + clear_dropped_attributes(tuple, reform); + slot = reform; + } + else + slot = src; - heap_insert(NewHeap, newtuple, GetCurrentCommandId(true), - HEAP_INSERT_NO_LOGICAL, bistate); + /* + * clear_dropped_attributes() should have deformed the tuple, so nothing + * should depend on it now. + */ + if (shouldFree) + heap_freetuple(tuple); - heap_freetuple(newtuple); + table_tuple_insert(rel, slot, GetCurrentCommandId(true), + TABLE_INSERT_NO_LOGICAL, bistate); } -/* - * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. - * - * Deform the given tuple, set values of dropped columns to NULL, and fill in - * any values from attmissingval; then form a new tuple and return it. If no - * attributes need to be changed, a copy of the original tuple is returned. - * Caller is responsible for freeing the returned tuple. - * - * XXX this coding assumes that both relations have the same tupledesc. - */ -static HeapTuple -reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap, - Datum *values, bool *isnull) +static bool +tuple_needs_reform(HeapTuple tuple, TupleDesc tupDesc) { - TupleDesc oldTupDesc = RelationGetDescr(OldHeap); - TupleDesc newTupDesc = RelationGetDescr(NewHeap); - bool needs_reform = false; - /* * A short tuple might require values from attmissing val, so activate the * coding unconditionally in that case. The value might legitimally be * NULL otherwise, so this is slightly wasteful, but it probably beats * having to test each attribute for presence of attmissingval each time. */ - if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts) - needs_reform = true; + if (HeapTupleHeaderGetNatts(tuple->t_data) < tupDesc->natts) + return true; - /* - * If the column has been dropped but a value is still present, we can - * optimize storage now by getting rid of it. - */ - if (!needs_reform) + /* Does it have dropped attributes? */ + for (int i = 0; i < tupDesc->natts; i++) { - for (int i = 0; i < newTupDesc->natts; i++) - { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped && - !heap_attisnull(tuple, i + 1, newTupDesc)) - { - needs_reform = true; - break; - } - } + if (TupleDescCompactAttr(tupDesc, i)->attisdropped && + !heap_attisnull(tuple, i + 1, tupDesc)) + return true; } - /* Skip work if no changes are needed */ - if (!needs_reform) - return heap_copytuple(tuple); + return false; +} - heap_deform_tuple(tuple, oldTupDesc, values, isnull); +/* + * Subroutine for reform_and_rewrite_tuple and heap_insert_for_repack. + * + * Set values of dropped columns to NULL, + */ +static void +clear_dropped_attributes(HeapTuple tuple, TupleTableSlot *reform) +{ + TupleDesc tupDesc = reform->tts_tupleDescriptor; + + /* Assuming 'reform' is virtual, this deforms the tuple. */ + Assert(TTS_IS_VIRTUAL(reform)); + ExecForceStoreHeapTuple(tuple, reform, false); - for (int i = 0; i < newTupDesc->natts; i++) + for (int i = 0; i < tupDesc->natts; i++) { - if (TupleDescCompactAttr(newTupDesc, i)->attisdropped) - isnull[i] = true; + if (TupleDescCompactAttr(tupDesc, i)->attisdropped) + reform->tts_isnull[i] = true; } - - return heap_form_tuple(newTupDesc, values, isnull); } /* -- 2.52.0 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v02-0002-Move-functions-to-repack.c.patch ^ permalink raw reply [nested|flat] 390+ messages in thread
end of thread, other threads:[~2026-07-15 08:37 UTC | newest] Thread overview: 390+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-05-14 23:26 [PATCH v19 6/8] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-06-16 11:54 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]> 2026-07-15 08:37 [PATCH 1/8] Use tuple slot to pass tuples for rewriting. Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox