agora inbox for [email protected]
help / color / mirror / Atom feedRe: "SELECT ... FROM DUAL" is not quite as silly as it appears
19+ messages / 6 participants
[nested] [flat]
* Re: "SELECT ... FROM DUAL" is not quite as silly as it appears
@ 2018-10-14 22:04 Tom Lane <[email protected]>
0 siblings, 1 reply; 19+ messages in thread
From: Tom Lane @ 2018-10-14 22:04 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Robert Haas <[email protected]> writes:
> On Thu, Mar 15, 2018 at 11:27 AM, Tom Lane <[email protected]> wrote:
>> We've long made fun of Oracle(TM) for the fact that if you just want
>> to evaluate some expressions, you have to write "select ... from dual"
>> rather than just "select ...". But I've realized recently that there's
>> a bit of method in that madness after all. Specifically, having to
>> cope with FromExprs that contain no base relation is fairly problematic
>> in the planner. prepjointree.c is basically unable to cope with
>> flattening a subquery that looks that way, although we've inserted a
>> lot of overly-baroque logic to handle some subsets of the case (cf
>> is_simple_subquery(), around line 1500). If memory serves, there are
>> other places that are complicated by the case.
>>
>> Suppose that, either in the rewriter or early in the planner, we were
>> to replace such cases with nonempty FromExprs, by adding a dummy RTE
>> representing a table with no columns and one row. This would in turn
>> give rise to an ordinary Path that converts to a Result plan, so that
>> the case is handled without any special contortions later. Then there
>> is no case where we don't have a nonempty relids set identifying a
>> subquery, so that all that special-case hackery in prepjointree.c
>> goes away, and we can simplify whatever else is having a hard time
>> with it.
> Good idea.
Here's a proposed patch along those lines. Some notes for review:
* The new RTE kind is "RTE_RESULT" after the kind of Plan node it will
produce. I'm not entirely in love with that name, but couldn't think
of a better idea.
* I renamed the existing ResultPath node type to GroupResultPath to
clarify that it's not used to scan RTE_RESULT RTEs, but just for
degenerate grouping cases. RTE_RESULT RTEs (usually) give rise to plain
Path nodes with pathtype T_Result. It doesn't work very well to try to
unify those two cases, even though they give rise to identical Plans,
because there are different rules about where the associated quals live
during query_planner.
* The interesting changes are in prepjointree.c; almost all the rest
of the patch is boilerplate to support RTE_RESULT RTEs in mostly the
same way that other special RTE-scan plan types are handled in the
planner. In prepjointree.c, I ended up getting rid of the original
decision to try to delete removable RTEs during pull_up_subqueries,
and instead made it happen in a separate traversal of the join tree.
That's a lot less complicated, and it has better results because we
can optimize more cases once we've seen the results of expression
preprocessing and outer-join strength reduction.
* I tried to get rid of the empty-jointree special case in query_planner
altogether. While the patch works fine that way, it makes for a
measurable slowdown in planning trivial queries --- I saw close to 10%
degradation in TPS rate for a pgbench test case that was just
$ cat trivialselect.sql
select 2+2;
$ pgbench -n -T 10 -f trivialselect.sql
So I ended up putting back the special case, but it's much less of a
cheat than it was before; the RelOptInfo it hands back is basically the
same as the normal path would produce. For me, the patch as given is
within the noise level of being the same speed as HEAD on this case.
* There's a hack in nodeResult.c to prevent the executor from crashing
on a whole-row Var for an RTE_RESULT RTE, which is something that the
planner will create in SELECT FOR UPDATE cases, because it thinks it
needs to provide a ROW_MARK_COPY image of the RTE's output. We might
be able to get rid of that if we could teach the planner that it need
not bother rowmarking RESULT RTEs, but that seemed like it would be
really messy. (At the point where the decision is made, we don't know
whether a subquery might end up as just a RESULT, or indeed vanish
entirely.) Since I couldn't measure any reproducible penalty from
having the extra setup cost for a Result plan, I left it like this.
* There are several existing test cases in join.sql whose plans change
for the better with this patch, so I didn't really think we needed any
additional test cases to show that it's working.
* There are a couple of cosmetic changes in EXPLAIN output as a result
of ruleutils.c seeing more RTEs in the plan's rtable than it did before,
causing it to decide to table-qualify Var names in more cases. We could
maybe spend more effort on ruleutils.c's heuristic for when to qualify,
but that seems like a separable problem, and anyway it's only cosmetic.
I'll add this to the next CF.
regards, tom lane
Attachments:
[text/x-diff] get-rid-of-empty-jointrees-1.patch (92.2K, ../../[email protected]/2-get-rid-of-empty-jointrees-1.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 33f9a79..efa5596 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2404,6 +2404,8 @@ JumbleRangeTable(pgssJumbleState *jstate, List *rtable)
case RTE_NAMEDTUPLESTORE:
APP_JUMB_STRING(rte->enrname);
break;
+ case RTE_RESULT:
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 21a2ef5..ac3722a 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5374,7 +5374,7 @@ INSERT INTO ft2 (c1,c2,c3) VALUES (1200,999,'foo') RETURNING tableoid::regclass;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Insert on public.ft2
- Output: (tableoid)::regclass
+ Output: (ft2.tableoid)::regclass
Remote SQL: INSERT INTO "S 1"."T 1"("C 1", c2, c3, c4, c5, c6, c7, c8) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
-> Result
Output: 1200, 999, NULL::integer, 'foo'::text, NULL::timestamp with time zone, NULL::timestamp without time zone, NULL::character varying, 'ft2 '::character(10), NULL::user_enum
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index 9e78421..8f7d4ba 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -437,9 +437,12 @@ ExecSupportsMarkRestore(Path *pathnode)
return ExecSupportsMarkRestore(((ProjectionPath *) pathnode)->subpath);
else if (IsA(pathnode, MinMaxAggPath))
return false; /* childless Result */
+ else if (IsA(pathnode, GroupResultPath))
+ return false; /* childless Result */
else
{
- Assert(IsA(pathnode, ResultPath));
+ /* Simple RTE_RESULT base relation */
+ Assert(IsA(pathnode, Path));
return false; /* childless Result */
}
diff --git a/src/backend/executor/nodeResult.c b/src/backend/executor/nodeResult.c
index e4418a2..f84e320 100644
--- a/src/backend/executor/nodeResult.c
+++ b/src/backend/executor/nodeResult.c
@@ -215,6 +215,21 @@ ExecInitResult(Result *node, EState *estate, int eflags)
Assert(innerPlan(node) == NULL);
/*
+ * If we're scanning an RTE_RESULT "relation", we might be asked to
+ * produce a whole-row Var representing the relation's current row. Set
+ * up a zero-column tuple in the scantuple slot to support this.
+ */
+ if (outerPlan(node) == NULL)
+ {
+ TupleDesc tupDesc;
+ TupleTableSlot *slot;
+
+ tupDesc = ExecTypeFromTL(NIL, false);
+ slot = ExecInitNullTupleSlot(estate, tupDesc);
+ resstate->ps.ps_ExprContext->ecxt_scantuple = slot;
+ }
+
+ /*
* Initialize result slot, type and projection.
*/
ExecInitResultTupleSlotTL(estate, &resstate->ps);
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index a10014f..ecaeeb3 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -2319,10 +2319,6 @@ range_table_walker(List *rtable,
if (walker(rte->tablesample, context))
return true;
break;
- case RTE_CTE:
- case RTE_NAMEDTUPLESTORE:
- /* nothing to do */
- break;
case RTE_SUBQUERY:
if (!(flags & QTW_IGNORE_RT_SUBQUERIES))
if (walker(rte->subquery, context))
@@ -2345,6 +2341,11 @@ range_table_walker(List *rtable,
if (walker(rte->values_lists, context))
return true;
break;
+ case RTE_CTE:
+ case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
+ /* nothing to do */
+ break;
}
if (walker(rte->securityQuals, context))
@@ -3150,10 +3151,6 @@ range_table_mutator(List *rtable,
TableSampleClause *);
/* we don't bother to copy eref, aliases, etc; OK? */
break;
- case RTE_CTE:
- case RTE_NAMEDTUPLESTORE:
- /* nothing to do */
- break;
case RTE_SUBQUERY:
if (!(flags & QTW_IGNORE_RT_SUBQUERIES))
{
@@ -3184,6 +3181,11 @@ range_table_mutator(List *rtable,
case RTE_VALUES:
MUTATE(newrte->values_lists, rte->values_lists, List *);
break;
+ case RTE_CTE:
+ case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
+ /* nothing to do */
+ break;
}
MUTATE(newrte->securityQuals, rte->securityQuals, List *);
newrt = lappend(newrt, newrte);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 69731cc..bd47989 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1954,9 +1954,9 @@ _outMergeAppendPath(StringInfo str, const MergeAppendPath *node)
}
static void
-_outResultPath(StringInfo str, const ResultPath *node)
+_outGroupResultPath(StringInfo str, const GroupResultPath *node)
{
- WRITE_NODE_TYPE("RESULTPATH");
+ WRITE_NODE_TYPE("GROUPRESULTPATH");
_outPathInfo(str, (const Path *) node);
@@ -2312,7 +2312,6 @@ _outPlannerInfo(StringInfo str, const PlannerInfo *node)
WRITE_ENUM_FIELD(inhTargetKind, InheritanceKind);
WRITE_BOOL_FIELD(hasJoinRTEs);
WRITE_BOOL_FIELD(hasLateralRTEs);
- WRITE_BOOL_FIELD(hasDeletedRTEs);
WRITE_BOOL_FIELD(hasHavingQual);
WRITE_BOOL_FIELD(hasPseudoConstantQuals);
WRITE_BOOL_FIELD(hasRecursion);
@@ -3167,6 +3166,9 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_NODE_FIELD(coltypmods);
WRITE_NODE_FIELD(colcollations);
break;
+ case RTE_RESULT:
+ /* no extra fields */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) node->rtekind);
break;
@@ -4055,8 +4057,8 @@ outNode(StringInfo str, const void *obj)
case T_MergeAppendPath:
_outMergeAppendPath(str, obj);
break;
- case T_ResultPath:
- _outResultPath(str, obj);
+ case T_GroupResultPath:
+ _outGroupResultPath(str, obj);
break;
case T_MaterialPath:
_outMaterialPath(str, obj);
diff --git a/src/backend/nodes/print.c b/src/backend/nodes/print.c
index b9bad5e..5a0500d 100644
--- a/src/backend/nodes/print.c
+++ b/src/backend/nodes/print.c
@@ -295,6 +295,10 @@ print_rt(const List *rtable)
printf("%d\t%s\t[tuplestore]",
i, rte->eref->aliasname);
break;
+ case RTE_RESULT:
+ printf("%d\t%s\t[result]",
+ i, rte->eref->aliasname);
+ break;
default:
printf("%d\t%s\t[unknown rtekind]",
i, rte->eref->aliasname);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index e117867..bd21829 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1410,6 +1410,9 @@ _readRangeTblEntry(void)
READ_NODE_FIELD(coltypmods);
READ_NODE_FIELD(colcollations);
break;
+ case RTE_RESULT:
+ /* no extra fields */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d",
(int) local_node->rtekind);
diff --git a/src/backend/optimizer/README b/src/backend/optimizer/README
index 9c852a1..89ce373 100644
--- a/src/backend/optimizer/README
+++ b/src/backend/optimizer/README
@@ -361,7 +361,16 @@ RelOptInfo - a relation or joined relations
join clauses)
Path - every way to generate a RelOptInfo(sequential,index,joins)
- SeqScan - represents a sequential scan plan
+ A plain Path node can represent several simple plans, per its pathtype:
+ T_SeqScan - sequential scan
+ T_SampleScan - tablesample scan
+ T_FunctionScan - function-in-FROM scan
+ T_TableFuncScan - table function scan
+ T_ValuesScan - VALUES scan
+ T_CteScan - CTE (WITH) scan
+ T_NamedTuplestoreScan - ENR scan
+ T_WorkTableScan - scan worktable of a recursive CTE
+ T_Result - childless Result plan node (used for FROM-less SELECT)
IndexPath - index scan
BitmapHeapPath - top of a bitmapped index scan
TidPath - scan by CTID
@@ -370,7 +379,7 @@ RelOptInfo - a relation or joined relations
CustomPath - for custom scan providers
AppendPath - append multiple subpaths together
MergeAppendPath - merge multiple subpaths, preserving their common sort order
- ResultPath - a childless Result plan node (used for FROM-less SELECT)
+ GroupResultPath - childless Result plan node (used for degenerate grouping)
MaterialPath - a Material plan node
UniquePath - remove duplicate rows (either by hashing or sorting)
GatherPath - collect the results of parallel workers
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 5f74d3b..26d5ed3 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -116,6 +116,8 @@ static void set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
static void set_namedtuplestore_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
+static void set_result_pathlist(PlannerInfo *root, RelOptInfo *rel,
+ RangeTblEntry *rte);
static void set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
static RelOptInfo *make_rel_from_joinlist(PlannerInfo *root, List *joinlist);
@@ -400,8 +402,13 @@ set_rel_size(PlannerInfo *root, RelOptInfo *rel,
set_cte_pathlist(root, rel, rte);
break;
case RTE_NAMEDTUPLESTORE:
+ /* Might as well just build the path immediately */
set_namedtuplestore_pathlist(root, rel, rte);
break;
+ case RTE_RESULT:
+ /* Might as well just build the path immediately */
+ set_result_pathlist(root, rel, rte);
+ break;
default:
elog(ERROR, "unexpected rtekind: %d", (int) rel->rtekind);
break;
@@ -473,6 +480,9 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
case RTE_NAMEDTUPLESTORE:
/* tuplestore reference --- fully handled during set_rel_size */
break;
+ case RTE_RESULT:
+ /* simple Result --- fully handled during set_rel_size */
+ break;
default:
elog(ERROR, "unexpected rtekind: %d", (int) rel->rtekind);
break;
@@ -675,6 +685,10 @@ set_rel_consider_parallel(PlannerInfo *root, RelOptInfo *rel,
* infrastructure to support that.
*/
return;
+
+ case RTE_RESULT:
+ /* Sure, execute it in a worker if you want. */
+ break;
}
/*
@@ -2468,6 +2482,36 @@ set_namedtuplestore_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * set_result_pathlist
+ * Build the (single) access path for an RTE_RESULT RTE
+ *
+ * There's no need for a separate set_result_size phase, since we
+ * don't support join-qual-parameterized paths for these RTEs.
+ */
+static void
+set_result_pathlist(PlannerInfo *root, RelOptInfo *rel,
+ RangeTblEntry *rte)
+{
+ Relids required_outer;
+
+ /* Mark rel with estimated output rows, width, etc */
+ set_result_size_estimates(root, rel);
+
+ /*
+ * We don't support pushing join clauses into the quals of a Result scan,
+ * but it could still have required parameterization due to LATERAL refs
+ * in its tlist.
+ */
+ required_outer = rel->lateral_relids;
+
+ /* Generate appropriate path */
+ add_path(rel, create_resultscan_path(root, rel, required_outer));
+
+ /* Select cheapest path (pretty easy in this case...) */
+ set_cheapest(rel);
+}
+
+/*
* set_worktable_pathlist
* Build the (single) access path for a self-reference CTE RTE
*
@@ -3635,9 +3679,6 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_SampleScan:
ptype = "SampleScan";
break;
- case T_SubqueryScan:
- ptype = "SubqueryScan";
- break;
case T_FunctionScan:
ptype = "FunctionScan";
break;
@@ -3650,6 +3691,12 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_CteScan:
ptype = "CteScan";
break;
+ case T_NamedTuplestoreScan:
+ ptype = "NamedTuplestoreScan";
+ break;
+ case T_Result:
+ ptype = "Result";
+ break;
case T_WorkTableScan:
ptype = "WorkTableScan";
break;
@@ -3674,7 +3721,7 @@ print_path(PlannerInfo *root, Path *path, int indent)
ptype = "TidScan";
break;
case T_SubqueryScanPath:
- ptype = "SubqueryScanScan";
+ ptype = "SubqueryScan";
break;
case T_ForeignPath:
ptype = "ForeignScan";
@@ -3700,8 +3747,8 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_MergeAppendPath:
ptype = "MergeAppend";
break;
- case T_ResultPath:
- ptype = "Result";
+ case T_GroupResultPath:
+ ptype = "GroupResult";
break;
case T_MaterialPath:
ptype = "Material";
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 7bf67a0..19f8e40 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1568,6 +1568,40 @@ cost_namedtuplestorescan(Path *path, PlannerInfo *root,
}
/*
+ * cost_resultscan
+ * Determines and returns the cost of scanning an RTE_RESULT relation.
+ */
+void
+cost_resultscan(Path *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info)
+{
+ Cost startup_cost = 0;
+ Cost run_cost = 0;
+ QualCost qpqual_cost;
+ Cost cpu_per_tuple;
+
+ /* Should only be applied to RTE_RESULT base relations */
+ Assert(baserel->relid > 0);
+ Assert(baserel->rtekind == RTE_RESULT);
+
+ /* Mark the path with the correct row estimate */
+ if (param_info)
+ path->rows = param_info->ppi_rows;
+ else
+ path->rows = baserel->rows;
+
+ /* We charge qual cost plus cpu_tuple_cost */
+ get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost);
+
+ startup_cost += qpqual_cost.startup;
+ cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple;
+ run_cost += cpu_per_tuple * baserel->tuples;
+
+ path->startup_cost = startup_cost;
+ path->total_cost = startup_cost + run_cost;
+}
+
+/*
* cost_recursive_union
* Determines and returns the cost of performing a recursive union,
* and also the estimated output size.
@@ -5037,6 +5071,32 @@ set_namedtuplestore_size_estimates(PlannerInfo *root, RelOptInfo *rel)
}
/*
+ * set_result_size_estimates
+ * Set the size estimates for an RTE_RESULT base relation
+ *
+ * The rel's targetlist and restrictinfo list must have been constructed
+ * already.
+ *
+ * We set the same fields as set_baserel_size_estimates.
+ */
+void
+set_result_size_estimates(PlannerInfo *root, RelOptInfo *rel)
+{
+ RangeTblEntry *rte PG_USED_FOR_ASSERTS_ONLY;
+
+ /* Should only be applied to RTE_RESULT base relations */
+ Assert(rel->relid > 0);
+ rte = planner_rt_fetch(rel->relid, root);
+ Assert(rte->rtekind == RTE_RESULT);
+
+ /* RTE_RESULT always generates a single row, natively */
+ rel->tuples = 1;
+
+ /* Now estimate number of output rows, etc */
+ set_baserel_size_estimates(root, rel);
+}
+
+/*
* set_foreign_size_estimates
* Set the size estimates for a base relation that is a foreign table.
*
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ae46b01..8f4d075 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -85,7 +85,8 @@ static Plan *create_gating_plan(PlannerInfo *root, Path *path, Plan *plan,
static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path);
static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path);
static Plan *create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path);
-static Result *create_result_plan(PlannerInfo *root, ResultPath *best_path);
+static Result *create_group_result_plan(PlannerInfo *root,
+ GroupResultPath *best_path);
static ProjectSet *create_project_set_plan(PlannerInfo *root, ProjectSetPath *best_path);
static Material *create_material_plan(PlannerInfo *root, MaterialPath *best_path,
int flags);
@@ -139,6 +140,8 @@ static CteScan *create_ctescan_plan(PlannerInfo *root, Path *best_path,
List *tlist, List *scan_clauses);
static NamedTuplestoreScan *create_namedtuplestorescan_plan(PlannerInfo *root,
Path *best_path, List *tlist, List *scan_clauses);
+static Result *create_resultscan_plan(PlannerInfo *root, Path *best_path,
+ List *tlist, List *scan_clauses);
static WorkTableScan *create_worktablescan_plan(PlannerInfo *root, Path *best_path,
List *tlist, List *scan_clauses);
static ForeignScan *create_foreignscan_plan(PlannerInfo *root, ForeignPath *best_path,
@@ -406,11 +409,16 @@ create_plan_recurse(PlannerInfo *root, Path *best_path, int flags)
plan = (Plan *) create_minmaxagg_plan(root,
(MinMaxAggPath *) best_path);
}
+ else if (IsA(best_path, GroupResultPath))
+ {
+ plan = (Plan *) create_group_result_plan(root,
+ (GroupResultPath *) best_path);
+ }
else
{
- Assert(IsA(best_path, ResultPath));
- plan = (Plan *) create_result_plan(root,
- (ResultPath *) best_path);
+ /* Simple RTE_RESULT base relation */
+ Assert(IsA(best_path, Path));
+ plan = create_scan_plan(root, best_path, flags);
}
break;
case T_ProjectSet:
@@ -694,6 +702,13 @@ create_scan_plan(PlannerInfo *root, Path *best_path, int flags)
scan_clauses);
break;
+ case T_Result:
+ plan = (Plan *) create_resultscan_plan(root,
+ best_path,
+ tlist,
+ scan_clauses);
+ break;
+
case T_WorkTableScan:
plan = (Plan *) create_worktablescan_plan(root,
best_path,
@@ -925,17 +940,34 @@ create_gating_plan(PlannerInfo *root, Path *path, Plan *plan,
List *gating_quals)
{
Plan *gplan;
+ Plan *splan;
Assert(gating_quals);
/*
+ * We might have a trivial Result plan already. Stacking one Result atop
+ * another is silly, so if that applies, just discard the input plan.
+ * (We're assuming its targetlist is uninteresting; it should be either
+ * the same as the result of build_path_tlist, or a simplified version.)
+ */
+ splan = plan;
+ if (IsA(plan, Result))
+ {
+ Result *rplan = (Result *) plan;
+
+ if (rplan->plan.lefttree == NULL &&
+ rplan->resconstantqual == NULL)
+ splan = NULL;
+ }
+
+ /*
* Since we need a Result node anyway, always return the path's requested
* tlist; that's never a wrong choice, even if the parent node didn't ask
* for CP_EXACT_TLIST.
*/
gplan = (Plan *) make_result(build_path_tlist(root, path),
(Node *) gating_quals,
- plan);
+ splan);
/*
* Notice that we don't change cost or size estimates when doing gating.
@@ -1257,15 +1289,14 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path)
}
/*
- * create_result_plan
+ * create_group_result_plan
* Create a Result plan for 'best_path'.
- * This is only used for degenerate cases, such as a query with an empty
- * jointree.
+ * This is only used for degenerate grouping cases.
*
* Returns a Plan node.
*/
static Result *
-create_result_plan(PlannerInfo *root, ResultPath *best_path)
+create_group_result_plan(PlannerInfo *root, GroupResultPath *best_path)
{
Result *plan;
List *tlist;
@@ -3412,6 +3443,44 @@ create_namedtuplestorescan_plan(PlannerInfo *root, Path *best_path,
}
/*
+ * create_resultscan_plan
+ * Returns a Result plan for the RTE_RESULT base relation scanned by
+ * 'best_path' with restriction clauses 'scan_clauses' and targetlist
+ * 'tlist'.
+ */
+static Result *
+create_resultscan_plan(PlannerInfo *root, Path *best_path,
+ List *tlist, List *scan_clauses)
+{
+ Result *scan_plan;
+ Index scan_relid = best_path->parent->relid;
+ RangeTblEntry *rte;
+
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RESULT);
+
+ /* Sort clauses into best execution order */
+ scan_clauses = order_qual_clauses(root, scan_clauses);
+
+ /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */
+ scan_clauses = extract_actual_clauses(scan_clauses, false);
+
+ /* Replace any outer-relation variables with nestloop params */
+ if (best_path->param_info)
+ {
+ scan_clauses = (List *)
+ replace_nestloop_params(root, (Node *) scan_clauses);
+ }
+
+ scan_plan = make_result(tlist, (Node *) scan_clauses, NULL);
+
+ copy_generic_path_info(&scan_plan->plan, best_path);
+
+ return scan_plan;
+}
+
+/*
* create_worktablescan_plan
* Returns a worktablescan plan for the base relation scanned by 'best_path'
* with restriction clauses 'scan_clauses' and targetlist 'tlist'.
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index 01335db..2d50d63 100644
--- a/src/backend/optimizer/plan/initsplan.c
+++ b/src/backend/optimizer/plan/initsplan.c
@@ -827,7 +827,7 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
* all below it, so we should report inner_join_rels = qualscope. If
* there was exactly one element, we should (and already did) report
* whatever its inner_join_rels were. If there were no elements (is
- * that possible?) the initialization before the loop fixed it.
+ * that still possible?) the initialization before the loop fixed it.
*/
if (list_length(f->fromlist) > 1)
*inner_join_rels = *qualscope;
diff --git a/src/backend/optimizer/plan/planmain.c b/src/backend/optimizer/plan/planmain.c
index b05adc7..20edfa4 100644
--- a/src/backend/optimizer/plan/planmain.c
+++ b/src/backend/optimizer/plan/planmain.c
@@ -61,44 +61,6 @@ query_planner(PlannerInfo *root, List *tlist,
double total_pages;
/*
- * If the query has an empty join tree, then it's something easy like
- * "SELECT 2+2;" or "INSERT ... VALUES()". Fall through quickly.
- */
- if (parse->jointree->fromlist == NIL)
- {
- /* We need a dummy joinrel to describe the empty set of baserels */
- final_rel = build_empty_join_rel(root);
-
- /*
- * If query allows parallelism in general, check whether the quals are
- * parallel-restricted. (We need not check final_rel->reltarget
- * because it's empty at this point. Anything parallel-restricted in
- * the query tlist will be dealt with later.)
- */
- if (root->glob->parallelModeOK)
- final_rel->consider_parallel =
- is_parallel_safe(root, parse->jointree->quals);
-
- /* The only path for it is a trivial Result path */
- add_path(final_rel, (Path *)
- create_result_path(root, final_rel,
- final_rel->reltarget,
- (List *) parse->jointree->quals));
-
- /* Select cheapest path (pretty easy in this case...) */
- set_cheapest(final_rel);
-
- /*
- * We still are required to call qp_callback, in case it's something
- * like "SELECT 2+2 ORDER BY 1".
- */
- root->canon_pathkeys = NIL;
- (*qp_callback) (root, qp_extra);
-
- return final_rel;
- }
-
- /*
* Init planner lists to empty.
*
* NOTE: append_rel_list was set up by subquery_planner, so do not touch
@@ -125,6 +87,71 @@ query_planner(PlannerInfo *root, List *tlist,
setup_simple_rel_arrays(root);
/*
+ * In the trivial case where the jointree is a single RTE_RESULT relation,
+ * bypass all the rest of this function and just make a RelOptInfo and its
+ * one access path. This is worth optimizing because it applies for
+ * common cases like "SELECT expression" and "INSERT ... VALUES()".
+ */
+ Assert(parse->jointree->fromlist != NIL);
+ if (list_length(parse->jointree->fromlist) == 1)
+ {
+ Node *jtnode = (Node *) linitial(parse->jointree->fromlist);
+
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int varno = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = root->simple_rte_array[varno];
+
+ Assert(rte != NULL);
+ if (rte->rtekind == RTE_RESULT)
+ {
+ /* Make the RelOptInfo for it directly */
+ final_rel = build_simple_rel(root, varno, NULL);
+
+ /*
+ * If query allows parallelism in general, check whether the
+ * quals are parallel-restricted. (We need not check
+ * final_rel->reltarget because it's empty at this point.
+ * Anything parallel-restricted in the query tlist will be
+ * dealt with later.) This is normally pretty silly, because
+ * a Result-only plan would never be interesting to
+ * parallelize. However, if force_parallel_mode is on, then
+ * we want to execute the Result in a parallel worker if
+ * possible, so we must do this.
+ */
+ if (root->glob->parallelModeOK &&
+ force_parallel_mode != FORCE_PARALLEL_OFF)
+ final_rel->consider_parallel =
+ is_parallel_safe(root, parse->jointree->quals);
+
+ /*
+ * The only path for it is a trivial Result path. We cheat a
+ * bit here by using a GroupResultPath, because that way we
+ * can just jam the quals into it without preprocessing them.
+ * (But, if you hold your head at the right angle, a FROM-less
+ * SELECT is a kind of degenerate-grouping case, so it's not
+ * that much of a cheat.)
+ */
+ add_path(final_rel, (Path *)
+ create_group_result_path(root, final_rel,
+ final_rel->reltarget,
+ (List *) parse->jointree->quals));
+
+ /* Select cheapest path (pretty easy in this case...) */
+ set_cheapest(final_rel);
+
+ /*
+ * We still are required to call qp_callback, in case it's
+ * something like "SELECT 2+2 ORDER BY 1".
+ */
+ (*qp_callback) (root, qp_extra);
+
+ return final_rel;
+ }
+ }
+ }
+
+ /*
* Populate append_rel_array with each AppendRelInfo to allow direct
* lookups by child relid.
*/
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index c729a99..ecc74a7 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -606,6 +606,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
List *newWithCheckOptions;
List *newHaving;
bool hasOuterJoins;
+ bool hasResultRTEs;
RelOptInfo *final_rel;
ListCell *l;
@@ -647,6 +648,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
SS_process_ctes(root);
/*
+ * If the FROM clause is empty, replace it with a dummy RTE_RESULT RTE, so
+ * that we don't need so many special cases to deal with that situation.
+ */
+ replace_empty_jointree(parse);
+
+ /*
* Look for ANY and EXISTS SubLinks in WHERE and JOIN/ON clauses, and try
* to transform them into joins. Note that this step does not descend
* into subqueries; if we pull up any subqueries below, their SubLinks are
@@ -679,14 +686,16 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
/*
* Detect whether any rangetable entries are RTE_JOIN kind; if not, we can
- * avoid the expense of doing flatten_join_alias_vars(). Also check for
- * outer joins --- if none, we can skip reduce_outer_joins(). And check
- * for LATERAL RTEs, too. This must be done after we have done
- * pull_up_subqueries(), of course.
+ * avoid the expense of doing flatten_join_alias_vars(). Likewise check
+ * whether any are RTE_RESULT kind; if not, we can skip
+ * remove_useless_result_rtes(). Also check for outer joins --- if none,
+ * we can skip reduce_outer_joins(). And check for LATERAL RTEs, too.
+ * This must be done after we have done pull_up_subqueries(), of course.
*/
root->hasJoinRTEs = false;
root->hasLateralRTEs = false;
hasOuterJoins = false;
+ hasResultRTEs = false;
foreach(l, parse->rtable)
{
RangeTblEntry *rte = lfirst_node(RangeTblEntry, l);
@@ -697,6 +706,10 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
if (IS_OUTER_JOIN(rte->jointype))
hasOuterJoins = true;
}
+ else if (rte->rtekind == RTE_RESULT)
+ {
+ hasResultRTEs = true;
+ }
if (rte->lateral)
root->hasLateralRTEs = true;
}
@@ -712,10 +725,10 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
/*
* Expand any rangetable entries that are inheritance sets into "append
* relations". This can add entries to the rangetable, but they must be
- * plain base relations not joins, so it's OK (and marginally more
- * efficient) to do it after checking for join RTEs. We must do it after
- * pulling up subqueries, else we'd fail to handle inherited tables in
- * subqueries.
+ * plain RTE_RELATION entries, so it's OK (and marginally more efficient)
+ * to do it after checking for joins and other special RTEs. We must do
+ * this after pulling up subqueries, else we'd fail to handle inherited
+ * tables in subqueries.
*/
expand_inherited_tables(root);
@@ -963,6 +976,14 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
reduce_outer_joins(root);
/*
+ * If we have any RTE_RESULT relations, see if they can be deleted from
+ * the jointree. This step is most effectively done after we've done
+ * expression preprocessing and outer join reduction.
+ */
+ if (hasResultRTEs)
+ remove_useless_result_rtes(root);
+
+ /*
* Do the main planning. If we have an inherited target relation, that
* needs special processing, else go straight to grouping_planner.
*/
@@ -3889,9 +3910,9 @@ create_degenerate_grouping_paths(PlannerInfo *root, RelOptInfo *input_rel,
while (--nrows >= 0)
{
path = (Path *)
- create_result_path(root, grouped_rel,
- grouped_rel->reltarget,
- (List *) parse->havingQual);
+ create_group_result_path(root, grouped_rel,
+ grouped_rel->reltarget,
+ (List *) parse->havingQual);
paths = lappend(paths, path);
}
path = (Path *)
@@ -3909,9 +3930,9 @@ create_degenerate_grouping_paths(PlannerInfo *root, RelOptInfo *input_rel,
{
/* No grouping sets, or just one, so one output row */
path = (Path *)
- create_result_path(root, grouped_rel,
- grouped_rel->reltarget,
- (List *) parse->havingQual);
+ create_group_result_path(root, grouped_rel,
+ grouped_rel->reltarget,
+ (List *) parse->havingQual);
}
add_path(grouped_rel, path);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7..39b70cf 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1460,12 +1460,6 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink,
return NULL;
/*
- * The subquery must have a nonempty jointree, else we won't have a join.
- */
- if (subselect->jointree->fromlist == NIL)
- return NULL;
-
- /*
* Separate out the WHERE clause. (We could theoretically also remove
* top-level plain JOIN/ON clauses, but it's probably not worth the
* trouble.)
@@ -1494,6 +1488,11 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink,
return NULL;
/*
+ * The subquery must have a nonempty jointree, but we can make it so.
+ */
+ replace_empty_jointree(subselect);
+
+ /*
* Prepare to pull up the sub-select into top range table.
*
* We rely here on the assumption that the outer query has no references
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index cd6e119..dc0c0f3 100644
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -4,12 +4,14 @@
* Planner preprocessing for subqueries and join tree manipulation.
*
* NOTE: the intended sequence for invoking these operations is
+ * replace_empty_jointree
* pull_up_sublinks
* inline_set_returning_functions
* pull_up_subqueries
* flatten_simple_union_all
* do expression preprocessing (including flattening JOIN alias vars)
* reduce_outer_joins
+ * remove_useless_result_rtes
*
*
* Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
@@ -66,14 +68,12 @@ static Node *pull_up_sublinks_qual_recurse(PlannerInfo *root, Node *node,
static Node *pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok);
+ AppendRelInfo *containing_appendrel);
static Node *pull_up_simple_subquery(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok);
+ AppendRelInfo *containing_appendrel);
static Node *pull_up_simple_union_all(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte);
static void pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root,
@@ -82,12 +82,10 @@ static void pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root,
static void make_setop_translation_list(Query *query, Index newvarno,
List **translated_vars);
static bool is_simple_subquery(Query *subquery, RangeTblEntry *rte,
- JoinExpr *lowest_outer_join,
- bool deletion_ok);
+ JoinExpr *lowest_outer_join);
static Node *pull_up_simple_values(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte);
-static bool is_simple_values(PlannerInfo *root, RangeTblEntry *rte,
- bool deletion_ok);
+static bool is_simple_values(PlannerInfo *root, RangeTblEntry *rte);
static bool is_simple_union_all(Query *subquery);
static bool is_simple_union_all_recurse(Node *setOp, Query *setOpQuery,
List *colTypes);
@@ -103,7 +101,6 @@ static Node *pullup_replace_vars_callback(Var *var,
replace_rte_variables_context *context);
static Query *pullup_replace_vars_subquery(Query *query,
pullup_replace_vars_context *context);
-static Node *pull_up_subqueries_cleanup(Node *jtnode);
static reduce_outer_joins_state *reduce_outer_joins_pass1(Node *jtnode);
static void reduce_outer_joins_pass2(Node *jtnode,
reduce_outer_joins_state *state,
@@ -111,14 +108,62 @@ static void reduce_outer_joins_pass2(Node *jtnode,
Relids nonnullable_rels,
List *nonnullable_vars,
List *forced_null_vars);
-static void substitute_multiple_relids(Node *node,
- int varno, Relids subrelids);
+static Node *remove_useless_results_recurse(PlannerInfo *root, Node *jtnode);
+static int is_result_ref(PlannerInfo *root, Node *jtnode);
+static void remove_result_refs(PlannerInfo *root, int varno, Node *newjtloc);
+static bool find_dependent_phvs(Node *node, int varno);
+static void substitute_phv_relids(Node *node,
+ int varno, Relids subrelids);
static void fix_append_rel_relids(List *append_rel_list, int varno,
Relids subrelids);
static Node *find_jointree_node_for_rel(Node *jtnode, int relid);
/*
+ * replace_empty_jointree
+ * If the Query's jointree is empty, replace it with a dummy RTE_RESULT
+ * relation.
+ *
+ * By doing this, we can avoid a bunch of corner cases that formerly existed
+ * for SELECTs with omitted FROM clauses. An example is that a subquery
+ * with empty jointree previously could not be pulled up, because that would
+ * have resulted in an empty relid set, making the subquery not uniquely
+ * identifiable for join or PlaceHolderVar processing.
+ *
+ * Unlike most other functions in this file, this function doesn't recurse;
+ * we rely on other processing to invoke it on sub-queries at suitable times.
+ */
+void
+replace_empty_jointree(Query *parse)
+{
+ RangeTblEntry *rte;
+ Index rti;
+ RangeTblRef *rtr;
+
+ /* Nothing to do if jointree is already nonempty */
+ if (parse->jointree->fromlist != NIL)
+ return;
+
+ /* We mustn't change it in the top level of a setop tree, either */
+ if (parse->setOperations)
+ return;
+
+ /* Create suitable RTE */
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RESULT;
+ rte->eref = makeAlias("*RESULT*", NIL);
+
+ /* Add it to rangetable */
+ parse->rtable = lappend(parse->rtable, rte);
+ rti = list_length(parse->rtable);
+
+ /* And jam a reference into the jointree */
+ rtr = makeNode(RangeTblRef);
+ rtr->rtindex = rti;
+ parse->jointree->fromlist = list_make1(rtr);
+}
+
+/*
* pull_up_sublinks
* Attempt to pull up ANY and EXISTS SubLinks to be treated as
* semijoins or anti-semijoins.
@@ -611,16 +656,11 @@ pull_up_subqueries(PlannerInfo *root)
{
/* Top level of jointree must always be a FromExpr */
Assert(IsA(root->parse->jointree, FromExpr));
- /* Reset flag saying we need a deletion cleanup pass */
- root->hasDeletedRTEs = false;
/* Recursion starts with no containing join nor appendrel */
root->parse->jointree = (FromExpr *)
pull_up_subqueries_recurse(root, (Node *) root->parse->jointree,
- NULL, NULL, NULL, false);
- /* Apply cleanup phase if necessary */
- if (root->hasDeletedRTEs)
- root->parse->jointree = (FromExpr *)
- pull_up_subqueries_cleanup((Node *) root->parse->jointree);
+ NULL, NULL, NULL);
+ /* We should still have a FromExpr */
Assert(IsA(root->parse->jointree, FromExpr));
}
@@ -629,8 +669,6 @@ pull_up_subqueries(PlannerInfo *root)
* Recursive guts of pull_up_subqueries.
*
* This recursively processes the jointree and returns a modified jointree.
- * Or, if it's valid to drop the current node from the jointree completely,
- * it returns NULL.
*
* If this jointree node is within either side of an outer join, then
* lowest_outer_join references the lowest such JoinExpr node; otherwise
@@ -647,37 +685,27 @@ pull_up_subqueries(PlannerInfo *root)
* This forces use of the PlaceHolderVar mechanism for all non-Var targetlist
* items, and puts some additional restrictions on what can be pulled up.
*
- * deletion_ok is true if the caller can cope with us returning NULL for a
- * deletable leaf node (for example, a VALUES RTE that could be pulled up).
- * If it's false, we'll avoid pullup in such cases.
- *
* A tricky aspect of this code is that if we pull up a subquery we have
* to replace Vars that reference the subquery's outputs throughout the
* parent query, including quals attached to jointree nodes above the one
- * we are currently processing! We handle this by being careful not to
- * change the jointree structure while recursing: no nodes other than leaf
- * RangeTblRef entries and entirely-empty FromExprs will be replaced or
- * deleted. Also, we can't turn pullup_replace_vars loose on the whole
- * jointree, because it'll return a mutated copy of the tree; we have to
+ * we are currently processing! We handle this by being careful to maintain
+ * validity of the jointree structure while recursing, in the following sense:
+ * whenever we recurse, all qual expressions in the tree must be reachable
+ * from the top level, in case the recursive call needs to modify them.
+ *
+ * Notice also that we can't turn pullup_replace_vars loose on the whole
+ * jointree, because it'd return a mutated copy of the tree; we have to
* invoke it just on the quals, instead. This behavior is what makes it
* reasonable to pass lowest_outer_join and lowest_nulling_outer_join as
* pointers rather than some more-indirect way of identifying the lowest
* OJs. Likewise, we don't replace append_rel_list members but only their
* substructure, so the containing_appendrel reference is safe to use.
- *
- * Because of the rule that no jointree nodes with substructure can be
- * replaced, we cannot fully handle the case of deleting nodes from the tree:
- * when we delete one child of a JoinExpr, we need to replace the JoinExpr
- * with a FromExpr, and that can't happen here. Instead, we set the
- * root->hasDeletedRTEs flag, which tells pull_up_subqueries() that an
- * additional pass over the tree is needed to clean up.
*/
static Node *
pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok)
+ AppendRelInfo *containing_appendrel)
{
Assert(jtnode != NULL);
if (IsA(jtnode, RangeTblRef))
@@ -693,15 +721,13 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
* unless is_safe_append_member says so.
*/
if (rte->rtekind == RTE_SUBQUERY &&
- is_simple_subquery(rte->subquery, rte,
- lowest_outer_join, deletion_ok) &&
+ is_simple_subquery(rte->subquery, rte, lowest_outer_join) &&
(containing_appendrel == NULL ||
is_safe_append_member(rte->subquery)))
return pull_up_simple_subquery(root, jtnode, rte,
lowest_outer_join,
lowest_nulling_outer_join,
- containing_appendrel,
- deletion_ok);
+ containing_appendrel);
/*
* Alternatively, is it a simple UNION ALL subquery? If so, flatten
@@ -725,7 +751,7 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
if (rte->rtekind == RTE_VALUES &&
lowest_outer_join == NULL &&
containing_appendrel == NULL &&
- is_simple_values(root, rte, deletion_ok))
+ is_simple_values(root, rte))
return pull_up_simple_values(root, jtnode, rte);
/* Otherwise, do nothing at this node. */
@@ -733,50 +759,16 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
else if (IsA(jtnode, FromExpr))
{
FromExpr *f = (FromExpr *) jtnode;
- bool have_undeleted_child = false;
ListCell *l;
Assert(containing_appendrel == NULL);
-
- /*
- * If the FromExpr has quals, it's not deletable even if its parent
- * would allow deletion.
- */
- if (f->quals)
- deletion_ok = false;
-
+ /* Recursively transform all the child nodes */
foreach(l, f->fromlist)
{
- /*
- * In a non-deletable FromExpr, we can allow deletion of child
- * nodes so long as at least one child remains; so it's okay
- * either if any previous child survives, or if there's more to
- * come. If all children are deletable in themselves, we'll force
- * the last one to remain unflattened.
- *
- * As a separate matter, we can allow deletion of all children of
- * the top-level FromExpr in a query, since that's a special case
- * anyway.
- */
- bool sub_deletion_ok = (deletion_ok ||
- have_undeleted_child ||
- lnext(l) != NULL ||
- f == root->parse->jointree);
-
lfirst(l) = pull_up_subqueries_recurse(root, lfirst(l),
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- sub_deletion_ok);
- if (lfirst(l) != NULL)
- have_undeleted_child = true;
- }
-
- if (deletion_ok && !have_undeleted_child)
- {
- /* OK to delete this FromExpr entirely */
- root->hasDeletedRTEs = true; /* probably is set already */
- return NULL;
+ NULL);
}
}
else if (IsA(jtnode, JoinExpr))
@@ -788,22 +780,14 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
switch (j->jointype)
{
case JOIN_INNER:
-
- /*
- * INNER JOIN can allow deletion of either child node, but not
- * both. So right child gets permission to delete only if
- * left child didn't get removed.
- */
j->larg = pull_up_subqueries_recurse(root, j->larg,
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- true);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- j->larg != NULL);
+ NULL);
break;
case JOIN_LEFT:
case JOIN_SEMI:
@@ -811,37 +795,31 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
lowest_nulling_outer_join,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
j,
- NULL,
- false);
+ NULL);
break;
case JOIN_FULL:
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
j,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
j,
- NULL,
- false);
+ NULL);
break;
case JOIN_RIGHT:
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
j,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
lowest_nulling_outer_join,
- NULL,
- false);
+ NULL);
break;
default:
elog(ERROR, "unrecognized join type: %d",
@@ -861,8 +839,8 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
*
* jtnode is a RangeTblRef that has been tentatively identified as a simple
* subquery by pull_up_subqueries. We return the replacement jointree node,
- * or NULL if the subquery can be deleted entirely, or jtnode itself if we
- * determine that the subquery can't be pulled up after all.
+ * or jtnode itself if we determine that the subquery can't be pulled up
+ * after all.
*
* rte is the RangeTblEntry referenced by jtnode. Remaining parameters are
* as for pull_up_subqueries_recurse.
@@ -871,8 +849,7 @@ static Node *
pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok)
+ AppendRelInfo *containing_appendrel)
{
Query *parse = root->parse;
int varno = ((RangeTblRef *) jtnode)->rtindex;
@@ -926,6 +903,12 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
Assert(subquery->cteList == NIL);
/*
+ * If the FROM clause is empty, replace it with a dummy RTE_RESULT RTE, so
+ * that we don't need so many special cases to deal with that situation.
+ */
+ replace_empty_jointree(subquery);
+
+ /*
* Pull up any SubLinks within the subquery's quals, so that we don't
* leave unoptimized SubLinks behind.
*/
@@ -957,8 +940,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
* easier just to keep this "if" looking the same as the one in
* pull_up_subqueries_recurse.
*/
- if (is_simple_subquery(subquery, rte,
- lowest_outer_join, deletion_ok) &&
+ if (is_simple_subquery(subquery, rte, lowest_outer_join) &&
(containing_appendrel == NULL || is_safe_append_member(subquery)))
{
/* good to go */
@@ -1159,6 +1141,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
case RTE_JOIN:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* these can't contain any lateral references */
break;
}
@@ -1195,7 +1178,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
Relids subrelids;
subrelids = get_relids_in_jointree((Node *) subquery->jointree, false);
- substitute_multiple_relids((Node *) parse, varno, subrelids);
+ substitute_phv_relids((Node *) parse, varno, subrelids);
fix_append_rel_relids(root->append_rel_list, varno, subrelids);
}
@@ -1235,17 +1218,14 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
/*
* Return the adjusted subquery jointree to replace the RangeTblRef entry
- * in parent's jointree; or, if we're flattening a subquery with empty
- * FROM list, return NULL to signal deletion of the subquery from the
- * parent jointree (and set hasDeletedRTEs to ensure cleanup later).
+ * in parent's jointree; or, if the FromExpr is degenerate, just return
+ * its single member.
*/
- if (subquery->jointree->fromlist == NIL)
- {
- Assert(deletion_ok);
- Assert(subquery->jointree->quals == NULL);
- root->hasDeletedRTEs = true;
- return NULL;
- }
+ Assert(IsA(subquery->jointree, FromExpr));
+ Assert(subquery->jointree->fromlist != NIL);
+ if (subquery->jointree->quals == NULL &&
+ list_length(subquery->jointree->fromlist) == 1)
+ return (Node *) linitial(subquery->jointree->fromlist);
return (Node *) subquery->jointree;
}
@@ -1381,7 +1361,7 @@ pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root, int parentRTindex,
rtr = makeNode(RangeTblRef);
rtr->rtindex = childRTindex;
(void) pull_up_subqueries_recurse(root, (Node *) rtr,
- NULL, NULL, appinfo, false);
+ NULL, NULL, appinfo);
}
else if (IsA(setOp, SetOperationStmt))
{
@@ -1436,12 +1416,10 @@ make_setop_translation_list(Query *query, Index newvarno,
* (Note subquery is not necessarily equal to rte->subquery; it could be a
* processed copy of that.)
* lowest_outer_join is the lowest outer join above the subquery, or NULL.
- * deletion_ok is true if it'd be okay to delete the subquery entirely.
*/
static bool
is_simple_subquery(Query *subquery, RangeTblEntry *rte,
- JoinExpr *lowest_outer_join,
- bool deletion_ok)
+ JoinExpr *lowest_outer_join)
{
/*
* Let's just make sure it's a valid subselect ...
@@ -1491,44 +1469,6 @@ is_simple_subquery(Query *subquery, RangeTblEntry *rte,
return false;
/*
- * Don't pull up a subquery with an empty jointree, unless it has no quals
- * and deletion_ok is true and we're not underneath an outer join.
- *
- * query_planner() will correctly generate a Result plan for a jointree
- * that's totally empty, but we can't cope with an empty FromExpr
- * appearing lower down in a jointree: we identify join rels via baserelid
- * sets, so we couldn't distinguish a join containing such a FromExpr from
- * one without it. We can only handle such cases if the place where the
- * subquery is linked is a FromExpr or inner JOIN that would still be
- * nonempty after removal of the subquery, so that it's still identifiable
- * via its contained baserelids. Safe contexts are signaled by
- * deletion_ok.
- *
- * But even in a safe context, we must keep the subquery if it has any
- * quals, because it's unclear where to put them in the upper query.
- *
- * Also, we must forbid pullup if such a subquery is underneath an outer
- * join, because then we might need to wrap its output columns with
- * PlaceHolderVars, and the PHVs would then have empty relid sets meaning
- * we couldn't tell where to evaluate them. (This test is separate from
- * the deletion_ok flag for possible future expansion: deletion_ok tells
- * whether the immediate parent site in the jointree could cope, not
- * whether we'd have PHV issues. It's possible this restriction could be
- * fixed by letting the PHVs use the relids of the parent jointree item,
- * but that complication is for another day.)
- *
- * Note that deletion of a subquery is also dependent on the check below
- * that its targetlist contains no set-returning functions. Deletion from
- * a FROM list or inner JOIN is okay only if the subquery must return
- * exactly one row.
- */
- if (subquery->jointree->fromlist == NIL &&
- (subquery->jointree->quals != NULL ||
- !deletion_ok ||
- lowest_outer_join != NULL))
- return false;
-
- /*
* If the subquery is LATERAL, check for pullup restrictions from that.
*/
if (rte->lateral)
@@ -1602,9 +1542,10 @@ is_simple_subquery(Query *subquery, RangeTblEntry *rte,
* Pull up a single simple VALUES RTE.
*
* jtnode is a RangeTblRef that has been identified as a simple VALUES RTE
- * by pull_up_subqueries. We always return NULL indicating that the RTE
- * can be deleted entirely (all failure cases should have been detected by
- * is_simple_values()).
+ * by pull_up_subqueries. We always return a RangeTblRef representing a
+ * RESULT RTE to replace it (all failure cases should have been detected by
+ * is_simple_values()). Actually, what we return is just jtnode, because
+ * we replace the VALUES RTE in the rangetable with the RESULT RTE.
*
* rte is the RangeTblEntry referenced by jtnode. Because of the limited
* possible usage of VALUES RTEs, we do not need the remaining parameters
@@ -1703,11 +1644,23 @@ pull_up_simple_values(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte)
Assert(root->placeholder_list == NIL);
/*
- * Return NULL to signal deletion of the VALUES RTE from the parent
- * jointree (and set hasDeletedRTEs to ensure cleanup later).
+ * Replace the VALUES RTE with a RESULT RTE. The VALUES RTE is the only
+ * rtable entry in the current query level, so this is easy.
*/
- root->hasDeletedRTEs = true;
- return NULL;
+ Assert(list_length(parse->rtable) == 1);
+
+ /* Create suitable RTE */
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RESULT;
+ rte->eref = makeAlias("*RESULT*", NIL);
+
+ /* Replace rangetable */
+ parse->rtable = list_make1(rte);
+
+ /* We could manufacture a new RangeTblRef, but the one we have is fine */
+ Assert(varno == 1);
+
+ return jtnode;
}
/*
@@ -1716,24 +1669,16 @@ pull_up_simple_values(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte)
* to pull up into the parent query.
*
* rte is the RTE_VALUES RangeTblEntry to check.
- * deletion_ok is true if it'd be okay to delete the VALUES RTE entirely.
*/
static bool
-is_simple_values(PlannerInfo *root, RangeTblEntry *rte, bool deletion_ok)
+is_simple_values(PlannerInfo *root, RangeTblEntry *rte)
{
Assert(rte->rtekind == RTE_VALUES);
/*
- * We can only pull up a VALUES RTE if deletion_ok is true. It's
- * basically the same case as a sub-select with empty FROM list; see
- * comments in is_simple_subquery().
- */
- if (!deletion_ok)
- return false;
-
- /*
- * Also, there must be exactly one VALUES list, else it's not semantically
- * correct to delete the VALUES RTE.
+ * There must be exactly one VALUES list, else it's not semantically
+ * correct to replace the VALUES RTE with a RESULT RTE, nor would we have
+ * a unique set of expressions to substitute into the parent query.
*/
if (list_length(rte->values_lists) != 1)
return false;
@@ -1746,8 +1691,8 @@ is_simple_values(PlannerInfo *root, RangeTblEntry *rte, bool deletion_ok)
/*
* Don't pull up a VALUES that contains any set-returning or volatile
- * functions. Again, the considerations here are basically identical to
- * restrictions on a subquery's targetlist.
+ * functions. The considerations here are basically identical to the
+ * restrictions on a pull-able subquery's targetlist.
*/
if (expression_returns_set((Node *) rte->values_lists) ||
contain_volatile_functions((Node *) rte->values_lists))
@@ -1850,7 +1795,9 @@ is_safe_append_member(Query *subquery)
/*
* It's only safe to pull up the child if its jointree contains exactly
* one RTE, else the AppendRelInfo data structure breaks. The one base RTE
- * could be buried in several levels of FromExpr, however.
+ * could be buried in several levels of FromExpr, however. Also, if the
+ * child's jointree is completely empty, we can pull up because
+ * pull_up_simple_subquery will insert a single RTE_RESULT RTE instead.
*
* Also, the child can't have any WHERE quals because there's no place to
* put them in an appendrel. (This is a bit annoying...) If we didn't
@@ -1859,6 +1806,11 @@ is_safe_append_member(Query *subquery)
* fix_append_rel_relids().
*/
jtnode = subquery->jointree;
+ Assert(IsA(jtnode, FromExpr));
+ /* Check the completely-empty case */
+ if (jtnode->fromlist == NIL && jtnode->quals == NULL)
+ return true;
+ /* Check the more general case */
while (IsA(jtnode, FromExpr))
{
if (jtnode->quals != NULL)
@@ -2014,6 +1966,7 @@ replace_vars_in_jointree(Node *jtnode,
case RTE_JOIN:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* these shouldn't be marked LATERAL */
Assert(false);
break;
@@ -2290,65 +2243,6 @@ pullup_replace_vars_subquery(Query *query,
NULL);
}
-/*
- * pull_up_subqueries_cleanup
- * Recursively fix up jointree after deletion of some subqueries.
- *
- * The jointree now contains some NULL subtrees, which we need to get rid of.
- * In a FromExpr, just rebuild the child-node list with null entries deleted.
- * In an inner JOIN, replace the JoinExpr node with a one-child FromExpr.
- */
-static Node *
-pull_up_subqueries_cleanup(Node *jtnode)
-{
- Assert(jtnode != NULL);
- if (IsA(jtnode, RangeTblRef))
- {
- /* Nothing to do at leaf nodes. */
- }
- else if (IsA(jtnode, FromExpr))
- {
- FromExpr *f = (FromExpr *) jtnode;
- List *newfrom = NIL;
- ListCell *l;
-
- foreach(l, f->fromlist)
- {
- Node *child = (Node *) lfirst(l);
-
- if (child == NULL)
- continue;
- child = pull_up_subqueries_cleanup(child);
- newfrom = lappend(newfrom, child);
- }
- f->fromlist = newfrom;
- }
- else if (IsA(jtnode, JoinExpr))
- {
- JoinExpr *j = (JoinExpr *) jtnode;
-
- if (j->larg)
- j->larg = pull_up_subqueries_cleanup(j->larg);
- if (j->rarg)
- j->rarg = pull_up_subqueries_cleanup(j->rarg);
- if (j->larg == NULL)
- {
- Assert(j->jointype == JOIN_INNER);
- Assert(j->rarg != NULL);
- return (Node *) makeFromExpr(list_make1(j->rarg), j->quals);
- }
- else if (j->rarg == NULL)
- {
- Assert(j->jointype == JOIN_INNER);
- return (Node *) makeFromExpr(list_make1(j->larg), j->quals);
- }
- }
- else
- elog(ERROR, "unrecognized node type: %d",
- (int) nodeTag(jtnode));
- return jtnode;
-}
-
/*
* flatten_simple_union_all
@@ -2858,9 +2752,387 @@ reduce_outer_joins_pass2(Node *jtnode,
(int) nodeTag(jtnode));
}
+
+/*
+ * remove_useless_result_rtes
+ * Attempt to remove RTE_RESULT RTEs from the join tree.
+ *
+ * We can remove RTE_RESULT entries from the join tree using the knowledge
+ * that RTE_RESULT returns exactly one row and has no output columns. Hence,
+ * if one is inner-joined to anything else, we can delete it. Optimizations
+ * are also possible for some outer-join cases, as detailed below.
+ *
+ * Some of these optimizations depend on recognizing empty (constant-true)
+ * quals for FromExprs and JoinExprs. That makes it useful to apply this
+ * optimization pass after expression preprocessing, since that will have
+ * eliminated constant-true quals, allowing more cases to be recognized as
+ * optimizable. What's more, the usual reason for an RTE_RESULT to be present
+ * is that we pulled up a subquery or VALUES clause, thus very possibly
+ * replacing Vars with constants, making it more likely that a qual can be
+ * reduced to constant true. Also, because some optimizations depend on
+ * the outer-join type, it's best to have done reduce_outer_joins() first.
+ *
+ * A PlaceHolderVar referencing an RTE_RESULT RTE poses an obstacle to this
+ * process: we must remove the RTE_RESULT's relid from the PHV's phrels, but
+ * we must not reduce the phrels set to empty. If that would happen, and
+ * the RTE_RESULT is an immediate child of an outer join, we have to give up
+ * and not remove the RTE_RESULT: there is noplace else to evaluate the
+ * PlaceHolderVar. (That is, in such cases the RTE_RESULT *does* have output
+ * columns.) But if the RTE_RESULT is an immediate child of an inner join,
+ * we can change the PlaceHolderVar's phrels so as to evaluate it at the
+ * inner join instead. This is OK because we really only care that PHVs are
+ * evaluated above or below the correct outer joins.
+ *
+ * We used to try to do this work as part of pull_up_subqueries() where the
+ * potentially-optimizable cases get introduced; but it's way simpler, and
+ * more effective, to do it separately.
+ */
+void
+remove_useless_result_rtes(PlannerInfo *root)
+{
+ /* Top level of jointree must always be a FromExpr */
+ Assert(IsA(root->parse->jointree, FromExpr));
+ /* Recurse ... */
+ root->parse->jointree = (FromExpr *)
+ remove_useless_results_recurse(root, (Node *) root->parse->jointree);
+ /* We should still have a FromExpr */
+ Assert(IsA(root->parse->jointree, FromExpr));
+}
+
+/*
+ * remove_useless_results_recurse
+ * Recursive guts of remove_useless_result_rtes.
+ *
+ * This recursively processes the jointree and returns a modified jointree.
+ */
+static Node *
+remove_useless_results_recurse(PlannerInfo *root, Node *jtnode)
+{
+ Assert(jtnode != NULL);
+ if (IsA(jtnode, RangeTblRef))
+ {
+ /* Can't immediately do anything with a RangeTblRef */
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ Relids result_relids = NULL;
+ ListCell *cell;
+ ListCell *prev;
+ ListCell *next;
+
+ /*
+ * We can drop RTE_RESULT rels from the fromlist so long as at least
+ * one child remains, since joining to a one-row table changes
+ * nothing. The easiest way to mechanize this rule is to modify the
+ * list in-place, using list_delete_cell.
+ */
+ prev = NULL;
+ for (cell = list_head(f->fromlist); cell; cell = next)
+ {
+ Node *child = (Node *) lfirst(cell);
+ int varno;
+
+ /* Recursively transform child ... */
+ child = remove_useless_results_recurse(root, child);
+ /* ... and stick it back into the tree */
+ lfirst(cell) = child;
+ next = lnext(cell);
+
+ /*
+ * If it's an RTE_RESULT with at least one sibling, we can drop
+ * it. We don't yet know what the inner join's final relid set
+ * will be, so postpone cleanup of PHVs etc till after this loop.
+ */
+ if (list_length(f->fromlist) > 1 &&
+ (varno = is_result_ref(root, child)) != 0)
+ {
+ f->fromlist = list_delete_cell(f->fromlist, cell, prev);
+ result_relids = bms_add_member(result_relids, varno);
+ }
+ else
+ prev = cell;
+ }
+
+ /*
+ * Clean up if we dropped any RTE_RESULT RTEs. This is a bit
+ * inefficient if there's more than one, but it seems better to
+ * optimize the support code for the single-relid case.
+ */
+ if (result_relids)
+ {
+ int varno;
+
+ while ((varno = bms_first_member(result_relids)) >= 0)
+ remove_result_refs(root, varno, (Node *) f);
+ }
+
+ /*
+ * If we're not at the top of the jointree, it's valid to simplify a
+ * degenerate FromExpr into its single child. (At the top, we must
+ * keep the FromExpr since Query.jointree is required to point to a
+ * FromExpr.)
+ */
+ if (f != root->parse->jointree &&
+ f->quals == NULL &&
+ list_length(f->fromlist) == 1)
+ return (Node *) linitial(f->fromlist);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+ int varno;
+
+ /* First, recurse */
+ j->larg = remove_useless_results_recurse(root, j->larg);
+ j->rarg = remove_useless_results_recurse(root, j->rarg);
+
+ /* Apply join-type-specific optimization rules */
+ switch (j->jointype)
+ {
+ case JOIN_INNER:
+
+ /*
+ * An inner join is equivalent to a FromExpr, so if either
+ * side was simplified to an RTE_RESULT rel, we can replace
+ * the join with a FromExpr with just the other side; and if
+ * the qual is empty (JOIN ON TRUE) then we can omit the
+ * FromExpr as well.
+ */
+ if ((varno = is_result_ref(root, j->larg)) != 0)
+ {
+ remove_result_refs(root, varno, j->rarg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->rarg), j->quals);
+ else
+ jtnode = j->rarg;
+ }
+ else if ((varno = is_result_ref(root, j->rarg)) != 0)
+ {
+ remove_result_refs(root, varno, j->larg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->larg), j->quals);
+ else
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_LEFT:
+
+ /*
+ * We can simplify this case if the RHS is an RTE_RESULT, with
+ * two different possibilities:
+ *
+ * If the qual is empty (JOIN ON TRUE), then the join can be
+ * strength-reduced to a plain inner join, since each LHS row
+ * necessarily has exactly one join partner. So we can always
+ * discard the RHS, much as in the JOIN_INNER case above.
+ *
+ * Otherwise, it's still true that each LHS row should be
+ * returned exactly once, and since the RHS returns no columns
+ * (unless there are PHVs that have to be evaluated there), we
+ * don't much care if it's null-extended or not. So in this
+ * case also, we can just ignore the qual and discard the left
+ * join.
+ */
+ if ((varno = is_result_ref(root, j->rarg)) != 0 &&
+ (j->quals == NULL ||
+ !find_dependent_phvs((Node *) root->parse, varno)))
+ {
+ remove_result_refs(root, varno, j->larg);
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_RIGHT:
+ /* Mirror-image of the JOIN_LEFT case */
+ if ((varno = is_result_ref(root, j->larg)) != 0 &&
+ (j->quals == NULL ||
+ !find_dependent_phvs((Node *) root->parse, varno)))
+ {
+ remove_result_refs(root, varno, j->rarg);
+ jtnode = j->rarg;
+ }
+ break;
+ case JOIN_SEMI:
+
+ /*
+ * We may simplify this case if the RHS is an RTE_RESULT; the
+ * join qual becomes effectively just a filter qual for the
+ * LHS, since we should either return the LHS row or not. For
+ * simplicity we inject the filter qual into a new FromExpr.
+ *
+ * However, we can't simplify if there are PHVs to evaluate at
+ * the RTE_RESULT ... but that's impossible isn't it?
+ */
+ if ((varno = is_result_ref(root, j->rarg)) != 0)
+ {
+ Assert(!find_dependent_phvs((Node *) root->parse, varno));
+ remove_result_refs(root, varno, j->larg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->larg), j->quals);
+ else
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_FULL:
+ case JOIN_ANTI:
+ /* We have no special smarts for these cases */
+ break;
+ default:
+ elog(ERROR, "unrecognized join type: %d",
+ (int) j->jointype);
+ break;
+ }
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d",
+ (int) nodeTag(jtnode));
+ return jtnode;
+}
+
+/*
+ * is_result_ref
+ * If jtnode is a RangeTblRef for an RTE_RESULT RTE, return its relid;
+ * otherwise return 0.
+ */
+static inline int
+is_result_ref(PlannerInfo *root, Node *jtnode)
+{
+ int varno;
+
+ if (!IsA(jtnode, RangeTblRef))
+ return 0;
+ varno = ((RangeTblRef *) jtnode)->rtindex;
+ if (rt_fetch(varno, root->parse->rtable)->rtekind != RTE_RESULT)
+ return 0;
+ return varno;
+}
+
+/*
+ * remove_result_refs
+ * Helper routine for dropping an unneeded RTE_RESULT RTE.
+ *
+ * This doesn't physically remove the RTE from the jointree, because that's
+ * more easily handled in remove_useless_results_recurse. What it does do
+ * is the necessary cleanup in the rest of the tree: adjust any PHVs that
+ * may reference the RTE, and get rid of any rowmark for it. Be sure to
+ * call this at a point where the jointree is valid (no disconnected nodes).
+ *
+ * Note that we don't need to process the append_rel_list, since RTEs
+ * referenced directly in the jointree won't be appendrel members.
+ *
+ * varno is the RTE_RESULT's relid.
+ * newjtloc is the jointree location at which any PHVs referencing the
+ * RTE_RESULT should be evaluated instead.
+ */
+static void
+remove_result_refs(PlannerInfo *root, int varno, Node *newjtloc)
+{
+ ListCell *cell;
+ ListCell *prev;
+ ListCell *next;
+
+ /* Fix up PlaceHolderVars as needed */
+ /* If there are no PHVs anywhere, we can skip this bit */
+ if (root->glob->lastPHId != 0)
+ {
+ Relids subrelids;
+
+ subrelids = get_relids_in_jointree(newjtloc, false);
+ Assert(!bms_is_empty(subrelids));
+ substitute_phv_relids((Node *) root->parse, varno, subrelids);
+ }
+
+ /* Remove any PlanRowMark referencing the RTE */
+ prev = NULL;
+ for (cell = list_head(root->rowMarks); cell; cell = next)
+ {
+ PlanRowMark *rc = (PlanRowMark *) lfirst(cell);
+
+ next = lnext(cell);
+ if (rc->rti == varno)
+ {
+ /* remove it from the list */
+ root->rowMarks = list_delete_cell(root->rowMarks, cell, prev);
+ /* there should be at most one, so we can stop looking */
+ break;
+ }
+ else
+ prev = cell;
+ }
+}
+
+
+/*
+ * find_dependent_phvs - are there any PlaceHolderVars whose relids are
+ * exactly the given varno?
+ */
+
+typedef struct
+{
+ Relids relids;
+ int sublevels_up;
+} find_dependent_phvs_context;
+
+static bool
+find_dependent_phvs_walker(Node *node,
+ find_dependent_phvs_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, PlaceHolderVar))
+ {
+ PlaceHolderVar *phv = (PlaceHolderVar *) node;
+
+ if (phv->phlevelsup == context->sublevels_up &&
+ bms_equal(context->relids, phv->phrels))
+ return true;
+ /* fall through to examine children */
+ }
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node,
+ find_dependent_phvs_walker,
+ (void *) context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ /* Shouldn't need to handle planner auxiliary nodes here */
+ Assert(!IsA(node, SpecialJoinInfo));
+ Assert(!IsA(node, AppendRelInfo));
+ Assert(!IsA(node, PlaceHolderInfo));
+ Assert(!IsA(node, MinMaxAggInfo));
+
+ return expression_tree_walker(node, find_dependent_phvs_walker,
+ (void *) context);
+}
+
+static bool
+find_dependent_phvs(Node *node, int varno)
+{
+ find_dependent_phvs_context context;
+
+ context.relids = bms_make_singleton(varno);
+ context.sublevels_up = 0;
+
+ /*
+ * Must be prepared to start with a Query or a bare expression tree.
+ */
+ return query_or_expression_tree_walker(node,
+ find_dependent_phvs_walker,
+ (void *) &context,
+ 0);
+}
+
/*
- * substitute_multiple_relids - adjust node relid sets after pulling up
- * a subquery
+ * substitute_phv_relids - adjust PlaceHolderVar relid sets after pulling up
+ * a subquery or removing an RTE_RESULT jointree item
*
* Find any PlaceHolderVar nodes in the given tree that reference the
* pulled-up relid, and change them to reference the replacement relid(s).
@@ -2876,11 +3148,11 @@ typedef struct
int varno;
int sublevels_up;
Relids subrelids;
-} substitute_multiple_relids_context;
+} substitute_phv_relids_context;
static bool
-substitute_multiple_relids_walker(Node *node,
- substitute_multiple_relids_context *context)
+substitute_phv_relids_walker(Node *node,
+ substitute_phv_relids_context *context)
{
if (node == NULL)
return false;
@@ -2895,6 +3167,8 @@ substitute_multiple_relids_walker(Node *node,
context->subrelids);
phv->phrels = bms_del_member(phv->phrels,
context->varno);
+ /* Assert we haven't broken the PHV */
+ Assert(!bms_is_empty(phv->phrels));
}
/* fall through to examine children */
}
@@ -2905,7 +3179,7 @@ substitute_multiple_relids_walker(Node *node,
context->sublevels_up++;
result = query_tree_walker((Query *) node,
- substitute_multiple_relids_walker,
+ substitute_phv_relids_walker,
(void *) context, 0);
context->sublevels_up--;
return result;
@@ -2916,14 +3190,14 @@ substitute_multiple_relids_walker(Node *node,
Assert(!IsA(node, PlaceHolderInfo));
Assert(!IsA(node, MinMaxAggInfo));
- return expression_tree_walker(node, substitute_multiple_relids_walker,
+ return expression_tree_walker(node, substitute_phv_relids_walker,
(void *) context);
}
static void
-substitute_multiple_relids(Node *node, int varno, Relids subrelids)
+substitute_phv_relids(Node *node, int varno, Relids subrelids)
{
- substitute_multiple_relids_context context;
+ substitute_phv_relids_context context;
context.varno = varno;
context.sublevels_up = 0;
@@ -2933,7 +3207,7 @@ substitute_multiple_relids(Node *node, int varno, Relids subrelids)
* Must be prepared to start with a Query or a bare expression tree.
*/
query_or_expression_tree_walker(node,
- substitute_multiple_relids_walker,
+ substitute_phv_relids_walker,
(void *) &context,
0);
}
@@ -2943,7 +3217,7 @@ substitute_multiple_relids(Node *node, int varno, Relids subrelids)
*
* When we pull up a subquery, any AppendRelInfo references to the subquery's
* RT index have to be replaced by the substituted relid (and there had better
- * be only one). We also need to apply substitute_multiple_relids to their
+ * be only one). We also need to apply substitute_phv_relids to their
* translated_vars lists, since those might contain PlaceHolderVars.
*
* We assume we may modify the AppendRelInfo nodes in-place.
@@ -2974,9 +3248,9 @@ fix_append_rel_relids(List *append_rel_list, int varno, Relids subrelids)
appinfo->child_relid = subvarno;
}
- /* Also finish fixups for its translated vars */
- substitute_multiple_relids((Node *) appinfo->translated_vars,
- varno, subrelids);
+ /* Also fix up any PHVs in its translated vars */
+ substitute_phv_relids((Node *) appinfo->translated_vars,
+ varno, subrelids);
}
}
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index ee6f4cd..23f9559 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1661,7 +1661,7 @@ contain_leaked_vars_walker(Node *node, void *context)
* find_nonnullable_vars() is that the tested conditions really are different:
* a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" does not prove
* that either v1 or v2 can't be NULL, but it does prove that the t1 row
- * as a whole can't be all-NULL.
+ * as a whole can't be all-NULL. Also, the behavior for PHVs is different.
*
* top_level is true while scanning top-level AND/OR structure; here, showing
* the result is either FALSE or NULL is good enough. top_level is false when
@@ -1847,7 +1847,24 @@ find_nonnullable_rels_walker(Node *node, bool top_level)
{
PlaceHolderVar *phv = (PlaceHolderVar *) node;
+ /*
+ * If the contained expression forces any rels non-nullable, so does
+ * the PHV.
+ */
result = find_nonnullable_rels_walker((Node *) phv->phexpr, top_level);
+
+ /*
+ * If the PHV's syntactic scope is exactly one rel, it will be forced
+ * to be evaluated at that rel, and so it will behave like a Var of
+ * that rel: if the rel's entire output goes to null, so will the PHV.
+ * (If the syntactic scope is a join, we know that the PHV will go to
+ * null if the whole join does; but that is AND semantics while we
+ * need OR semantics for find_nonnullable_rels' result, so we can't do
+ * anything with the knowledge.)
+ */
+ if (phv->phlevelsup == 0 &&
+ bms_membership(phv->phrels) == BMS_SINGLETON)
+ result = bms_add_members(result, phv->phrels);
}
return result;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d50d86b..b48c958 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1429,17 +1429,17 @@ create_merge_append_path(PlannerInfo *root,
}
/*
- * create_result_path
+ * create_group_result_path
* Creates a path representing a Result-and-nothing-else plan.
*
- * This is only used for degenerate cases, such as a query with an empty
- * jointree.
+ * This is only used for degenerate grouping cases, in which we know we
+ * need to produce one result row, possibly filtered by a HAVING qual.
*/
-ResultPath *
-create_result_path(PlannerInfo *root, RelOptInfo *rel,
- PathTarget *target, List *resconstantqual)
+GroupResultPath *
+create_group_result_path(PlannerInfo *root, RelOptInfo *rel,
+ PathTarget *target, List *havingqual)
{
- ResultPath *pathnode = makeNode(ResultPath);
+ GroupResultPath *pathnode = makeNode(GroupResultPath);
pathnode->path.pathtype = T_Result;
pathnode->path.parent = rel;
@@ -1449,9 +1449,13 @@ create_result_path(PlannerInfo *root, RelOptInfo *rel,
pathnode->path.parallel_safe = rel->consider_parallel;
pathnode->path.parallel_workers = 0;
pathnode->path.pathkeys = NIL;
- pathnode->quals = resconstantqual;
+ pathnode->quals = havingqual;
- /* Hardly worth defining a cost_result() function ... just do it */
+ /*
+ * We can't quite use cost_resultscan() because the quals we want to
+ * account for are not baserestrict quals of the rel. Might as well just
+ * hack it here.
+ */
pathnode->path.rows = 1;
pathnode->path.startup_cost = target->cost.startup;
pathnode->path.total_cost = target->cost.startup +
@@ -1461,12 +1465,12 @@ create_result_path(PlannerInfo *root, RelOptInfo *rel,
* Add cost of qual, if any --- but we ignore its selectivity, since our
* rowcount estimate should be 1 no matter what the qual is.
*/
- if (resconstantqual)
+ if (havingqual)
{
QualCost qual_cost;
- cost_qual_eval(&qual_cost, resconstantqual, root);
- /* resconstantqual is evaluated once at startup */
+ cost_qual_eval(&qual_cost, havingqual, root);
+ /* havingqual is evaluated once at startup */
pathnode->path.startup_cost += qual_cost.startup + qual_cost.per_tuple;
pathnode->path.total_cost += qual_cost.startup + qual_cost.per_tuple;
}
@@ -2020,6 +2024,32 @@ create_namedtuplestorescan_path(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * create_resultscan_path
+ * Creates a path corresponding to a scan of an RTE_RESULT relation,
+ * returning the pathnode.
+ */
+Path *
+create_resultscan_path(PlannerInfo *root, RelOptInfo *rel,
+ Relids required_outer)
+{
+ Path *pathnode = makeNode(Path);
+
+ pathnode->pathtype = T_Result;
+ pathnode->parent = rel;
+ pathnode->pathtarget = rel->reltarget;
+ pathnode->param_info = get_baserel_parampathinfo(root, rel,
+ required_outer);
+ pathnode->parallel_aware = false;
+ pathnode->parallel_safe = rel->consider_parallel;
+ pathnode->parallel_workers = 0;
+ pathnode->pathkeys = NIL; /* result is always unordered */
+
+ cost_resultscan(pathnode, root, rel, pathnode->param_info);
+
+ return pathnode;
+}
+
+/*
* create_worktablescan_path
* Creates a path corresponding to a scan of a self-reference CTE,
* returning the pathnode.
@@ -3559,6 +3589,11 @@ reparameterize_path(PlannerInfo *root, Path *path,
spath->path.pathkeys,
required_outer);
}
+ case T_Result:
+ /* Supported only for RTE_RESULT scan paths */
+ if (IsA(path, Path))
+ return create_resultscan_path(root, rel, required_outer);
+ break;
case T_Append:
{
AppendPath *apath = (AppendPath *) path;
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8369e3a..2196189 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1629,6 +1629,7 @@ build_physical_tlist(PlannerInfo *root, RelOptInfo *rel)
case RTE_VALUES:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
NULL, &colvars);
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index 39f5729..cacf876 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -232,10 +232,12 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent)
case RTE_VALUES:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* Subquery, function, tablefunc, values list, CTE, or ENR --- set
- * up attr range and arrays
+ * up attr range and arrays. RTE_RESULT has no columns, but for
+ * simplicity process it here too.
*
* Note: 0 is included in range to support whole-row Vars
*/
@@ -1108,36 +1110,6 @@ subbuild_joinrel_joinlist(RelOptInfo *joinrel,
/*
- * build_empty_join_rel
- * Build a dummy join relation describing an empty set of base rels.
- *
- * This is used for queries with empty FROM clauses, such as "SELECT 2+2" or
- * "INSERT INTO foo VALUES(...)". We don't try very hard to make the empty
- * joinrel completely valid, since no real planning will be done with it ---
- * we just need it to carry a simple Result path out of query_planner().
- */
-RelOptInfo *
-build_empty_join_rel(PlannerInfo *root)
-{
- RelOptInfo *joinrel;
-
- /* The dummy join relation should be the only one ... */
- Assert(root->join_rel_list == NIL);
-
- joinrel = makeNode(RelOptInfo);
- joinrel->reloptkind = RELOPT_JOINREL;
- joinrel->relids = NULL; /* empty set */
- joinrel->rows = 1; /* we produce one row for such cases */
- joinrel->rtekind = RTE_JOIN;
- joinrel->reltarget = create_empty_pathtarget();
-
- root->join_rel_list = lappend(root->join_rel_list, joinrel);
-
- return joinrel;
-}
-
-
-/*
* fetch_upper_rel
* Build a RelOptInfo describing some post-scan/join query processing,
* or return a pre-existing one if somebody already built it.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 226927b..1d01256 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -2878,6 +2878,9 @@ transformLockingClause(ParseState *pstate, Query *qry, LockingClause *lc,
LCS_asString(lc->strength)),
parser_errposition(pstate, thisrel->location)));
break;
+
+ /* Shouldn't be possible to see RTE_RESULT here */
+
default:
elog(ERROR, "unrecognized RTE type: %d",
(int) rte->rtekind);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index f115ed8..f6413f3 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -2517,6 +2517,9 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
}
}
break;
+ case RTE_RESULT:
+ /* These expose no columns, so nothing to do */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
}
@@ -2909,6 +2912,14 @@ get_rte_attribute_type(RangeTblEntry *rte, AttrNumber attnum,
rte->eref->aliasname)));
}
break;
+ case RTE_RESULT:
+ /* this probably can't happen ... */
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column %d of relation \"%s\" does not exist",
+ attnum,
+ rte->eref->aliasname)));
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
}
@@ -3037,6 +3048,15 @@ get_rte_attribute_is_dropped(RangeTblEntry *rte, AttrNumber attnum)
result = false; /* keep compiler quiet */
}
break;
+ case RTE_RESULT:
+ /* this probably can't happen ... */
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column %d of relation \"%s\" does not exist",
+ attnum,
+ rte->eref->aliasname)));
+ result = false; /* keep compiler quiet */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
result = false; /* keep compiler quiet */
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 4932e58..8801ee0 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -398,6 +398,7 @@ markTargetListOrigin(ParseState *pstate, TargetEntry *tle,
case RTE_VALUES:
case RTE_TABLEFUNC:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* not a simple relation, leave it unmarked */
break;
case RTE_CTE:
@@ -1525,6 +1526,7 @@ expandRecordVariable(ParseState *pstate, Var *var, int levelsup)
case RTE_RELATION:
case RTE_VALUES:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* This case should not occur: a column of a table, values list,
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index f6693ea..d1ca1aa 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -7009,6 +7009,7 @@ get_name_for_var_field(Var *var, int fieldno,
case RTE_RELATION:
case RTE_VALUES:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* This case should not occur: a column of a table, values list,
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index cac6ff0..67a201a 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -237,7 +237,7 @@ typedef enum NodeTag
T_HashPath,
T_AppendPath,
T_MergeAppendPath,
- T_ResultPath,
+ T_GroupResultPath,
T_MaterialPath,
T_UniquePath,
T_GatherPath,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index aa4a0db..2fdcf67 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -951,7 +951,8 @@ typedef enum RTEKind
RTE_TABLEFUNC, /* TableFunc(.., column list) */
RTE_VALUES, /* VALUES (<exprlist>), (<exprlist>), ... */
RTE_CTE, /* common table expr (WITH list element) */
- RTE_NAMEDTUPLESTORE /* tuplestore, e.g. for AFTER triggers */
+ RTE_NAMEDTUPLESTORE, /* tuplestore, e.g. for AFTER triggers */
+ RTE_RESULT /* RTE represents an omitted FROM clause */
} RTEKind;
typedef struct RangeTblEntry
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 88d3723..359b044 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -323,7 +323,6 @@ typedef struct PlannerInfo
* partitioned table */
bool hasJoinRTEs; /* true if any RTEs are RTE_JOIN kind */
bool hasLateralRTEs; /* true if any RTEs are marked LATERAL */
- bool hasDeletedRTEs; /* true if any RTE was deleted from jointree */
bool hasHavingQual; /* true if havingQual was non-null */
bool hasPseudoConstantQuals; /* true if any RestrictInfo has
* pseudoconstant = true */
@@ -1345,17 +1343,17 @@ typedef struct MergeAppendPath
} MergeAppendPath;
/*
- * ResultPath represents use of a Result plan node to compute a variable-free
- * targetlist with no underlying tables (a "SELECT expressions" query).
- * The query could have a WHERE clause, too, represented by "quals".
+ * GroupResultPath represents use of a Result plan node to compute the
+ * output of a degenerate GROUP BY case, wherein we know we should produce
+ * exactly one row, which might then be filtered by a HAVING qual.
*
* Note that quals is a list of bare clauses, not RestrictInfos.
*/
-typedef struct ResultPath
+typedef struct GroupResultPath
{
Path path;
List *quals;
-} ResultPath;
+} GroupResultPath;
/*
* MaterialPath represents use of a Material plan node, i.e., caching of
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 77ca7ff..023733f 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -105,6 +105,8 @@ extern void cost_ctescan(Path *path, PlannerInfo *root,
RelOptInfo *baserel, ParamPathInfo *param_info);
extern void cost_namedtuplestorescan(Path *path, PlannerInfo *root,
RelOptInfo *baserel, ParamPathInfo *param_info);
+extern void cost_resultscan(Path *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info);
extern void cost_recursive_union(Path *runion, Path *nrterm, Path *rterm);
extern void cost_sort(Path *path, PlannerInfo *root,
List *pathkeys, Cost input_cost, double tuples, int width,
@@ -196,6 +198,7 @@ extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
double cte_rows);
extern void set_tablefunc_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern void set_namedtuplestore_size_estimates(PlannerInfo *root, RelOptInfo *rel);
+extern void set_result_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern void set_foreign_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern PathTarget *set_pathtarget_cost_width(PlannerInfo *root, PathTarget *target);
extern double compute_bitmap_pages(PlannerInfo *root, RelOptInfo *baserel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 81abcf5..87b6c50 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -75,8 +75,10 @@ extern MergeAppendPath *create_merge_append_path(PlannerInfo *root,
List *pathkeys,
Relids required_outer,
List *partitioned_rels);
-extern ResultPath *create_result_path(PlannerInfo *root, RelOptInfo *rel,
- PathTarget *target, List *resconstantqual);
+extern GroupResultPath *create_group_result_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ PathTarget *target,
+ List *havingqual);
extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
Path *subpath, SpecialJoinInfo *sjinfo);
@@ -105,6 +107,8 @@ extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
extern Path *create_namedtuplestorescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
+extern Path *create_resultscan_path(PlannerInfo *root, RelOptInfo *rel,
+ Relids required_outer);
extern Path *create_worktablescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
@@ -275,7 +279,6 @@ extern Relids min_join_parameterization(PlannerInfo *root,
Relids joinrelids,
RelOptInfo *outer_rel,
RelOptInfo *inner_rel);
-extern RelOptInfo *build_empty_join_rel(PlannerInfo *root);
extern RelOptInfo *fetch_upper_rel(PlannerInfo *root, UpperRelationKind kind,
Relids relids);
extern Relids find_childrel_parents(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/prep.h b/src/include/optimizer/prep.h
index 3860877..31c6d74 100644
--- a/src/include/optimizer/prep.h
+++ b/src/include/optimizer/prep.h
@@ -21,11 +21,13 @@
/*
* prototypes for prepjointree.c
*/
+extern void replace_empty_jointree(Query *parse);
extern void pull_up_sublinks(PlannerInfo *root);
extern void inline_set_returning_functions(PlannerInfo *root);
extern void pull_up_subqueries(PlannerInfo *root);
extern void flatten_simple_union_all(PlannerInfo *root);
extern void reduce_outer_joins(PlannerInfo *root);
+extern void remove_useless_result_rtes(PlannerInfo *root);
extern Relids get_relids_in_jointree(Node *jtnode, bool include_joins);
extern Relids get_relids_for_join(PlannerInfo *root, int joinrelid);
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 1f53780..de060bb 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -2227,20 +2227,17 @@ explain (costs off)
select * from int8_tbl i1 left join (int8_tbl i2 join
(select 123 as x) ss on i2.q1 = x) on i1.q2 = i2.q2
order by 1, 2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-------------------------------------------
Sort
Sort Key: i1.q1, i1.q2
-> Hash Left Join
Hash Cond: (i1.q2 = i2.q2)
-> Seq Scan on int8_tbl i1
-> Hash
- -> Hash Join
- Hash Cond: (i2.q1 = (123))
- -> Seq Scan on int8_tbl i2
- -> Hash
- -> Result
-(11 rows)
+ -> Seq Scan on int8_tbl i2
+ Filter: (q1 = 123)
+(8 rows)
select * from int8_tbl i1 left join (int8_tbl i2 join
(select 123 as x) ss on i2.q1 = x) on i1.q2 = i2.q2
@@ -3141,23 +3138,18 @@ select t1.unique2, t1.stringu1, t2.unique1, t2.stringu2 from
left join tenk1 t2
on (subq1.y1 = t2.unique1)
where t1.unique2 < 42 and t1.stringu1 > t2.stringu2;
- QUERY PLAN
------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------
Nested Loop
Join Filter: (t1.stringu1 > t2.stringu2)
-> Nested Loop
- Join Filter: ((0) = i1.f1)
- -> Nested Loop
- -> Nested Loop
- Join Filter: ((1) = (1))
- -> Result
- -> Result
- -> Index Scan using tenk1_unique2 on tenk1 t1
- Index Cond: ((unique2 = (11)) AND (unique2 < 42))
-> Seq Scan on int4_tbl i1
+ Filter: (f1 = 0)
+ -> Index Scan using tenk1_unique2 on tenk1 t1
+ Index Cond: ((unique2 = (11)) AND (unique2 < 42))
-> Index Scan using tenk1_unique1 on tenk1 t2
Index Cond: (unique1 = (3))
-(14 rows)
+(9 rows)
select t1.unique2, t1.stringu1, t2.unique1, t2.stringu2 from
tenk1 t1
@@ -3596,7 +3588,7 @@ select t1.* from
-> Hash Right Join
Output: i8.q2
Hash Cond: ((NULL::integer) = i8b1.q2)
- -> Hash Left Join
+ -> Hash Join
Output: i8.q2, (NULL::integer)
Hash Cond: (i8.q1 = i8b2.q1)
-> Seq Scan on public.int8_tbl i8
@@ -4018,10 +4010,10 @@ select * from
QUERY PLAN
---------------------------------------
Nested Loop Left Join
- Join Filter: ((1) = COALESCE((1)))
-> Result
-> Hash Full Join
Hash Cond: (a1.unique1 = (1))
+ Filter: (1 = COALESCE((1)))
-> Seq Scan on tenk1 a1
-> Hash
-> Result
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069..a54b4a5 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -999,7 +999,7 @@ select * from
QUERY PLAN
----------------------------------------------------------
Subquery Scan on ss
- Output: x, u
+ Output: ss.x, ss.u
Filter: tattle(ss.x, 8)
-> ProjectSet
Output: 9, unnest('{1,2,3,11,12,13}'::integer[])
@@ -1061,7 +1061,7 @@ select * from
QUERY PLAN
----------------------------------------------------------
Subquery Scan on ss
- Output: x, u
+ Output: ss.x, ss.u
Filter: tattle(ss.x, ss.u)
-> ProjectSet
Output: 9, unnest('{1,2,3,11,12,13}'::integer[])
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: "SELECT ... FROM DUAL" is not quite as silly as it appears
@ 2018-10-15 03:25 Tom Lane <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Tom Lane @ 2018-10-15 03:25 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
I wrote:
> * There's a hack in nodeResult.c to prevent the executor from crashing
> on a whole-row Var for an RTE_RESULT RTE, which is something that the
> planner will create in SELECT FOR UPDATE cases, because it thinks it
> needs to provide a ROW_MARK_COPY image of the RTE's output. We might
> be able to get rid of that if we could teach the planner that it need
> not bother rowmarking RESULT RTEs, but that seemed like it would be
> really messy. (At the point where the decision is made, we don't know
> whether a subquery might end up as just a RESULT, or indeed vanish
> entirely.) Since I couldn't measure any reproducible penalty from
> having the extra setup cost for a Result plan, I left it like this.
Well, I'd barely sent this when I realized that there was a better way.
The nodeResult.c hack predates my decision to postpone cleaning up
RTE_RESULT RTEs till near the end of the preprocessing phase, and
given that code, it is easy to get rid of rowmarking RESULT RTEs ...
in fact, the code was doing it already, except in the edge case of
a SELECT with only a RESULT RTE. So here's a version that does not
touch nodeResult.c.
regards, tom lane
Attachments:
[text/x-diff] get-rid-of-empty-jointrees-2.patch (92.4K, ../../[email protected]/2-get-rid-of-empty-jointrees-2.patch)
download | inline diff:
diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c
index 33f9a79..efa5596 100644
--- a/contrib/pg_stat_statements/pg_stat_statements.c
+++ b/contrib/pg_stat_statements/pg_stat_statements.c
@@ -2404,6 +2404,8 @@ JumbleRangeTable(pgssJumbleState *jstate, List *rtable)
case RTE_NAMEDTUPLESTORE:
APP_JUMB_STRING(rte->enrname);
break;
+ case RTE_RESULT:
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
break;
diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out
index 21a2ef5..ac3722a 100644
--- a/contrib/postgres_fdw/expected/postgres_fdw.out
+++ b/contrib/postgres_fdw/expected/postgres_fdw.out
@@ -5374,7 +5374,7 @@ INSERT INTO ft2 (c1,c2,c3) VALUES (1200,999,'foo') RETURNING tableoid::regclass;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Insert on public.ft2
- Output: (tableoid)::regclass
+ Output: (ft2.tableoid)::regclass
Remote SQL: INSERT INTO "S 1"."T 1"("C 1", c2, c3, c4, c5, c6, c7, c8) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
-> Result
Output: 1200, 999, NULL::integer, 'foo'::text, NULL::timestamp with time zone, NULL::timestamp without time zone, NULL::character varying, 'ft2 '::character(10), NULL::user_enum
diff --git a/src/backend/executor/execAmi.c b/src/backend/executor/execAmi.c
index 9e78421..8f7d4ba 100644
--- a/src/backend/executor/execAmi.c
+++ b/src/backend/executor/execAmi.c
@@ -437,9 +437,12 @@ ExecSupportsMarkRestore(Path *pathnode)
return ExecSupportsMarkRestore(((ProjectionPath *) pathnode)->subpath);
else if (IsA(pathnode, MinMaxAggPath))
return false; /* childless Result */
+ else if (IsA(pathnode, GroupResultPath))
+ return false; /* childless Result */
else
{
- Assert(IsA(pathnode, ResultPath));
+ /* Simple RTE_RESULT base relation */
+ Assert(IsA(pathnode, Path));
return false; /* childless Result */
}
diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c
index a10014f..ecaeeb3 100644
--- a/src/backend/nodes/nodeFuncs.c
+++ b/src/backend/nodes/nodeFuncs.c
@@ -2319,10 +2319,6 @@ range_table_walker(List *rtable,
if (walker(rte->tablesample, context))
return true;
break;
- case RTE_CTE:
- case RTE_NAMEDTUPLESTORE:
- /* nothing to do */
- break;
case RTE_SUBQUERY:
if (!(flags & QTW_IGNORE_RT_SUBQUERIES))
if (walker(rte->subquery, context))
@@ -2345,6 +2341,11 @@ range_table_walker(List *rtable,
if (walker(rte->values_lists, context))
return true;
break;
+ case RTE_CTE:
+ case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
+ /* nothing to do */
+ break;
}
if (walker(rte->securityQuals, context))
@@ -3150,10 +3151,6 @@ range_table_mutator(List *rtable,
TableSampleClause *);
/* we don't bother to copy eref, aliases, etc; OK? */
break;
- case RTE_CTE:
- case RTE_NAMEDTUPLESTORE:
- /* nothing to do */
- break;
case RTE_SUBQUERY:
if (!(flags & QTW_IGNORE_RT_SUBQUERIES))
{
@@ -3184,6 +3181,11 @@ range_table_mutator(List *rtable,
case RTE_VALUES:
MUTATE(newrte->values_lists, rte->values_lists, List *);
break;
+ case RTE_CTE:
+ case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
+ /* nothing to do */
+ break;
}
MUTATE(newrte->securityQuals, rte->securityQuals, List *);
newrt = lappend(newrt, newrte);
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 69731cc..bd47989 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1954,9 +1954,9 @@ _outMergeAppendPath(StringInfo str, const MergeAppendPath *node)
}
static void
-_outResultPath(StringInfo str, const ResultPath *node)
+_outGroupResultPath(StringInfo str, const GroupResultPath *node)
{
- WRITE_NODE_TYPE("RESULTPATH");
+ WRITE_NODE_TYPE("GROUPRESULTPATH");
_outPathInfo(str, (const Path *) node);
@@ -2312,7 +2312,6 @@ _outPlannerInfo(StringInfo str, const PlannerInfo *node)
WRITE_ENUM_FIELD(inhTargetKind, InheritanceKind);
WRITE_BOOL_FIELD(hasJoinRTEs);
WRITE_BOOL_FIELD(hasLateralRTEs);
- WRITE_BOOL_FIELD(hasDeletedRTEs);
WRITE_BOOL_FIELD(hasHavingQual);
WRITE_BOOL_FIELD(hasPseudoConstantQuals);
WRITE_BOOL_FIELD(hasRecursion);
@@ -3167,6 +3166,9 @@ _outRangeTblEntry(StringInfo str, const RangeTblEntry *node)
WRITE_NODE_FIELD(coltypmods);
WRITE_NODE_FIELD(colcollations);
break;
+ case RTE_RESULT:
+ /* no extra fields */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) node->rtekind);
break;
@@ -4055,8 +4057,8 @@ outNode(StringInfo str, const void *obj)
case T_MergeAppendPath:
_outMergeAppendPath(str, obj);
break;
- case T_ResultPath:
- _outResultPath(str, obj);
+ case T_GroupResultPath:
+ _outGroupResultPath(str, obj);
break;
case T_MaterialPath:
_outMaterialPath(str, obj);
diff --git a/src/backend/nodes/print.c b/src/backend/nodes/print.c
index b9bad5e..5a0500d 100644
--- a/src/backend/nodes/print.c
+++ b/src/backend/nodes/print.c
@@ -295,6 +295,10 @@ print_rt(const List *rtable)
printf("%d\t%s\t[tuplestore]",
i, rte->eref->aliasname);
break;
+ case RTE_RESULT:
+ printf("%d\t%s\t[result]",
+ i, rte->eref->aliasname);
+ break;
default:
printf("%d\t%s\t[unknown rtekind]",
i, rte->eref->aliasname);
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index e117867..bd21829 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1410,6 +1410,9 @@ _readRangeTblEntry(void)
READ_NODE_FIELD(coltypmods);
READ_NODE_FIELD(colcollations);
break;
+ case RTE_RESULT:
+ /* no extra fields */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d",
(int) local_node->rtekind);
diff --git a/src/backend/optimizer/README b/src/backend/optimizer/README
index 9c852a1..89ce373 100644
--- a/src/backend/optimizer/README
+++ b/src/backend/optimizer/README
@@ -361,7 +361,16 @@ RelOptInfo - a relation or joined relations
join clauses)
Path - every way to generate a RelOptInfo(sequential,index,joins)
- SeqScan - represents a sequential scan plan
+ A plain Path node can represent several simple plans, per its pathtype:
+ T_SeqScan - sequential scan
+ T_SampleScan - tablesample scan
+ T_FunctionScan - function-in-FROM scan
+ T_TableFuncScan - table function scan
+ T_ValuesScan - VALUES scan
+ T_CteScan - CTE (WITH) scan
+ T_NamedTuplestoreScan - ENR scan
+ T_WorkTableScan - scan worktable of a recursive CTE
+ T_Result - childless Result plan node (used for FROM-less SELECT)
IndexPath - index scan
BitmapHeapPath - top of a bitmapped index scan
TidPath - scan by CTID
@@ -370,7 +379,7 @@ RelOptInfo - a relation or joined relations
CustomPath - for custom scan providers
AppendPath - append multiple subpaths together
MergeAppendPath - merge multiple subpaths, preserving their common sort order
- ResultPath - a childless Result plan node (used for FROM-less SELECT)
+ GroupResultPath - childless Result plan node (used for degenerate grouping)
MaterialPath - a Material plan node
UniquePath - remove duplicate rows (either by hashing or sorting)
GatherPath - collect the results of parallel workers
diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c
index 5f74d3b..26d5ed3 100644
--- a/src/backend/optimizer/path/allpaths.c
+++ b/src/backend/optimizer/path/allpaths.c
@@ -116,6 +116,8 @@ static void set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
static void set_namedtuplestore_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
+static void set_result_pathlist(PlannerInfo *root, RelOptInfo *rel,
+ RangeTblEntry *rte);
static void set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel,
RangeTblEntry *rte);
static RelOptInfo *make_rel_from_joinlist(PlannerInfo *root, List *joinlist);
@@ -400,8 +402,13 @@ set_rel_size(PlannerInfo *root, RelOptInfo *rel,
set_cte_pathlist(root, rel, rte);
break;
case RTE_NAMEDTUPLESTORE:
+ /* Might as well just build the path immediately */
set_namedtuplestore_pathlist(root, rel, rte);
break;
+ case RTE_RESULT:
+ /* Might as well just build the path immediately */
+ set_result_pathlist(root, rel, rte);
+ break;
default:
elog(ERROR, "unexpected rtekind: %d", (int) rel->rtekind);
break;
@@ -473,6 +480,9 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel,
case RTE_NAMEDTUPLESTORE:
/* tuplestore reference --- fully handled during set_rel_size */
break;
+ case RTE_RESULT:
+ /* simple Result --- fully handled during set_rel_size */
+ break;
default:
elog(ERROR, "unexpected rtekind: %d", (int) rel->rtekind);
break;
@@ -675,6 +685,10 @@ set_rel_consider_parallel(PlannerInfo *root, RelOptInfo *rel,
* infrastructure to support that.
*/
return;
+
+ case RTE_RESULT:
+ /* Sure, execute it in a worker if you want. */
+ break;
}
/*
@@ -2468,6 +2482,36 @@ set_namedtuplestore_pathlist(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * set_result_pathlist
+ * Build the (single) access path for an RTE_RESULT RTE
+ *
+ * There's no need for a separate set_result_size phase, since we
+ * don't support join-qual-parameterized paths for these RTEs.
+ */
+static void
+set_result_pathlist(PlannerInfo *root, RelOptInfo *rel,
+ RangeTblEntry *rte)
+{
+ Relids required_outer;
+
+ /* Mark rel with estimated output rows, width, etc */
+ set_result_size_estimates(root, rel);
+
+ /*
+ * We don't support pushing join clauses into the quals of a Result scan,
+ * but it could still have required parameterization due to LATERAL refs
+ * in its tlist.
+ */
+ required_outer = rel->lateral_relids;
+
+ /* Generate appropriate path */
+ add_path(rel, create_resultscan_path(root, rel, required_outer));
+
+ /* Select cheapest path (pretty easy in this case...) */
+ set_cheapest(rel);
+}
+
+/*
* set_worktable_pathlist
* Build the (single) access path for a self-reference CTE RTE
*
@@ -3635,9 +3679,6 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_SampleScan:
ptype = "SampleScan";
break;
- case T_SubqueryScan:
- ptype = "SubqueryScan";
- break;
case T_FunctionScan:
ptype = "FunctionScan";
break;
@@ -3650,6 +3691,12 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_CteScan:
ptype = "CteScan";
break;
+ case T_NamedTuplestoreScan:
+ ptype = "NamedTuplestoreScan";
+ break;
+ case T_Result:
+ ptype = "Result";
+ break;
case T_WorkTableScan:
ptype = "WorkTableScan";
break;
@@ -3674,7 +3721,7 @@ print_path(PlannerInfo *root, Path *path, int indent)
ptype = "TidScan";
break;
case T_SubqueryScanPath:
- ptype = "SubqueryScanScan";
+ ptype = "SubqueryScan";
break;
case T_ForeignPath:
ptype = "ForeignScan";
@@ -3700,8 +3747,8 @@ print_path(PlannerInfo *root, Path *path, int indent)
case T_MergeAppendPath:
ptype = "MergeAppend";
break;
- case T_ResultPath:
- ptype = "Result";
+ case T_GroupResultPath:
+ ptype = "GroupResult";
break;
case T_MaterialPath:
ptype = "Material";
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 7bf67a0..19f8e40 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -1568,6 +1568,40 @@ cost_namedtuplestorescan(Path *path, PlannerInfo *root,
}
/*
+ * cost_resultscan
+ * Determines and returns the cost of scanning an RTE_RESULT relation.
+ */
+void
+cost_resultscan(Path *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info)
+{
+ Cost startup_cost = 0;
+ Cost run_cost = 0;
+ QualCost qpqual_cost;
+ Cost cpu_per_tuple;
+
+ /* Should only be applied to RTE_RESULT base relations */
+ Assert(baserel->relid > 0);
+ Assert(baserel->rtekind == RTE_RESULT);
+
+ /* Mark the path with the correct row estimate */
+ if (param_info)
+ path->rows = param_info->ppi_rows;
+ else
+ path->rows = baserel->rows;
+
+ /* We charge qual cost plus cpu_tuple_cost */
+ get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost);
+
+ startup_cost += qpqual_cost.startup;
+ cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple;
+ run_cost += cpu_per_tuple * baserel->tuples;
+
+ path->startup_cost = startup_cost;
+ path->total_cost = startup_cost + run_cost;
+}
+
+/*
* cost_recursive_union
* Determines and returns the cost of performing a recursive union,
* and also the estimated output size.
@@ -5037,6 +5071,32 @@ set_namedtuplestore_size_estimates(PlannerInfo *root, RelOptInfo *rel)
}
/*
+ * set_result_size_estimates
+ * Set the size estimates for an RTE_RESULT base relation
+ *
+ * The rel's targetlist and restrictinfo list must have been constructed
+ * already.
+ *
+ * We set the same fields as set_baserel_size_estimates.
+ */
+void
+set_result_size_estimates(PlannerInfo *root, RelOptInfo *rel)
+{
+ RangeTblEntry *rte PG_USED_FOR_ASSERTS_ONLY;
+
+ /* Should only be applied to RTE_RESULT base relations */
+ Assert(rel->relid > 0);
+ rte = planner_rt_fetch(rel->relid, root);
+ Assert(rte->rtekind == RTE_RESULT);
+
+ /* RTE_RESULT always generates a single row, natively */
+ rel->tuples = 1;
+
+ /* Now estimate number of output rows, etc */
+ set_baserel_size_estimates(root, rel);
+}
+
+/*
* set_foreign_size_estimates
* Set the size estimates for a base relation that is a foreign table.
*
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index ae46b01..8f4d075 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -85,7 +85,8 @@ static Plan *create_gating_plan(PlannerInfo *root, Path *path, Plan *plan,
static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path);
static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path);
static Plan *create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path);
-static Result *create_result_plan(PlannerInfo *root, ResultPath *best_path);
+static Result *create_group_result_plan(PlannerInfo *root,
+ GroupResultPath *best_path);
static ProjectSet *create_project_set_plan(PlannerInfo *root, ProjectSetPath *best_path);
static Material *create_material_plan(PlannerInfo *root, MaterialPath *best_path,
int flags);
@@ -139,6 +140,8 @@ static CteScan *create_ctescan_plan(PlannerInfo *root, Path *best_path,
List *tlist, List *scan_clauses);
static NamedTuplestoreScan *create_namedtuplestorescan_plan(PlannerInfo *root,
Path *best_path, List *tlist, List *scan_clauses);
+static Result *create_resultscan_plan(PlannerInfo *root, Path *best_path,
+ List *tlist, List *scan_clauses);
static WorkTableScan *create_worktablescan_plan(PlannerInfo *root, Path *best_path,
List *tlist, List *scan_clauses);
static ForeignScan *create_foreignscan_plan(PlannerInfo *root, ForeignPath *best_path,
@@ -406,11 +409,16 @@ create_plan_recurse(PlannerInfo *root, Path *best_path, int flags)
plan = (Plan *) create_minmaxagg_plan(root,
(MinMaxAggPath *) best_path);
}
+ else if (IsA(best_path, GroupResultPath))
+ {
+ plan = (Plan *) create_group_result_plan(root,
+ (GroupResultPath *) best_path);
+ }
else
{
- Assert(IsA(best_path, ResultPath));
- plan = (Plan *) create_result_plan(root,
- (ResultPath *) best_path);
+ /* Simple RTE_RESULT base relation */
+ Assert(IsA(best_path, Path));
+ plan = create_scan_plan(root, best_path, flags);
}
break;
case T_ProjectSet:
@@ -694,6 +702,13 @@ create_scan_plan(PlannerInfo *root, Path *best_path, int flags)
scan_clauses);
break;
+ case T_Result:
+ plan = (Plan *) create_resultscan_plan(root,
+ best_path,
+ tlist,
+ scan_clauses);
+ break;
+
case T_WorkTableScan:
plan = (Plan *) create_worktablescan_plan(root,
best_path,
@@ -925,17 +940,34 @@ create_gating_plan(PlannerInfo *root, Path *path, Plan *plan,
List *gating_quals)
{
Plan *gplan;
+ Plan *splan;
Assert(gating_quals);
/*
+ * We might have a trivial Result plan already. Stacking one Result atop
+ * another is silly, so if that applies, just discard the input plan.
+ * (We're assuming its targetlist is uninteresting; it should be either
+ * the same as the result of build_path_tlist, or a simplified version.)
+ */
+ splan = plan;
+ if (IsA(plan, Result))
+ {
+ Result *rplan = (Result *) plan;
+
+ if (rplan->plan.lefttree == NULL &&
+ rplan->resconstantqual == NULL)
+ splan = NULL;
+ }
+
+ /*
* Since we need a Result node anyway, always return the path's requested
* tlist; that's never a wrong choice, even if the parent node didn't ask
* for CP_EXACT_TLIST.
*/
gplan = (Plan *) make_result(build_path_tlist(root, path),
(Node *) gating_quals,
- plan);
+ splan);
/*
* Notice that we don't change cost or size estimates when doing gating.
@@ -1257,15 +1289,14 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path)
}
/*
- * create_result_plan
+ * create_group_result_plan
* Create a Result plan for 'best_path'.
- * This is only used for degenerate cases, such as a query with an empty
- * jointree.
+ * This is only used for degenerate grouping cases.
*
* Returns a Plan node.
*/
static Result *
-create_result_plan(PlannerInfo *root, ResultPath *best_path)
+create_group_result_plan(PlannerInfo *root, GroupResultPath *best_path)
{
Result *plan;
List *tlist;
@@ -3412,6 +3443,44 @@ create_namedtuplestorescan_plan(PlannerInfo *root, Path *best_path,
}
/*
+ * create_resultscan_plan
+ * Returns a Result plan for the RTE_RESULT base relation scanned by
+ * 'best_path' with restriction clauses 'scan_clauses' and targetlist
+ * 'tlist'.
+ */
+static Result *
+create_resultscan_plan(PlannerInfo *root, Path *best_path,
+ List *tlist, List *scan_clauses)
+{
+ Result *scan_plan;
+ Index scan_relid = best_path->parent->relid;
+ RangeTblEntry *rte;
+
+ Assert(scan_relid > 0);
+ rte = planner_rt_fetch(scan_relid, root);
+ Assert(rte->rtekind == RTE_RESULT);
+
+ /* Sort clauses into best execution order */
+ scan_clauses = order_qual_clauses(root, scan_clauses);
+
+ /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */
+ scan_clauses = extract_actual_clauses(scan_clauses, false);
+
+ /* Replace any outer-relation variables with nestloop params */
+ if (best_path->param_info)
+ {
+ scan_clauses = (List *)
+ replace_nestloop_params(root, (Node *) scan_clauses);
+ }
+
+ scan_plan = make_result(tlist, (Node *) scan_clauses, NULL);
+
+ copy_generic_path_info(&scan_plan->plan, best_path);
+
+ return scan_plan;
+}
+
+/*
* create_worktablescan_plan
* Returns a worktablescan plan for the base relation scanned by 'best_path'
* with restriction clauses 'scan_clauses' and targetlist 'tlist'.
diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c
index 01335db..2d50d63 100644
--- a/src/backend/optimizer/plan/initsplan.c
+++ b/src/backend/optimizer/plan/initsplan.c
@@ -827,7 +827,7 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join,
* all below it, so we should report inner_join_rels = qualscope. If
* there was exactly one element, we should (and already did) report
* whatever its inner_join_rels were. If there were no elements (is
- * that possible?) the initialization before the loop fixed it.
+ * that still possible?) the initialization before the loop fixed it.
*/
if (list_length(f->fromlist) > 1)
*inner_join_rels = *qualscope;
diff --git a/src/backend/optimizer/plan/planmain.c b/src/backend/optimizer/plan/planmain.c
index b05adc7..20edfa4 100644
--- a/src/backend/optimizer/plan/planmain.c
+++ b/src/backend/optimizer/plan/planmain.c
@@ -61,44 +61,6 @@ query_planner(PlannerInfo *root, List *tlist,
double total_pages;
/*
- * If the query has an empty join tree, then it's something easy like
- * "SELECT 2+2;" or "INSERT ... VALUES()". Fall through quickly.
- */
- if (parse->jointree->fromlist == NIL)
- {
- /* We need a dummy joinrel to describe the empty set of baserels */
- final_rel = build_empty_join_rel(root);
-
- /*
- * If query allows parallelism in general, check whether the quals are
- * parallel-restricted. (We need not check final_rel->reltarget
- * because it's empty at this point. Anything parallel-restricted in
- * the query tlist will be dealt with later.)
- */
- if (root->glob->parallelModeOK)
- final_rel->consider_parallel =
- is_parallel_safe(root, parse->jointree->quals);
-
- /* The only path for it is a trivial Result path */
- add_path(final_rel, (Path *)
- create_result_path(root, final_rel,
- final_rel->reltarget,
- (List *) parse->jointree->quals));
-
- /* Select cheapest path (pretty easy in this case...) */
- set_cheapest(final_rel);
-
- /*
- * We still are required to call qp_callback, in case it's something
- * like "SELECT 2+2 ORDER BY 1".
- */
- root->canon_pathkeys = NIL;
- (*qp_callback) (root, qp_extra);
-
- return final_rel;
- }
-
- /*
* Init planner lists to empty.
*
* NOTE: append_rel_list was set up by subquery_planner, so do not touch
@@ -125,6 +87,71 @@ query_planner(PlannerInfo *root, List *tlist,
setup_simple_rel_arrays(root);
/*
+ * In the trivial case where the jointree is a single RTE_RESULT relation,
+ * bypass all the rest of this function and just make a RelOptInfo and its
+ * one access path. This is worth optimizing because it applies for
+ * common cases like "SELECT expression" and "INSERT ... VALUES()".
+ */
+ Assert(parse->jointree->fromlist != NIL);
+ if (list_length(parse->jointree->fromlist) == 1)
+ {
+ Node *jtnode = (Node *) linitial(parse->jointree->fromlist);
+
+ if (IsA(jtnode, RangeTblRef))
+ {
+ int varno = ((RangeTblRef *) jtnode)->rtindex;
+ RangeTblEntry *rte = root->simple_rte_array[varno];
+
+ Assert(rte != NULL);
+ if (rte->rtekind == RTE_RESULT)
+ {
+ /* Make the RelOptInfo for it directly */
+ final_rel = build_simple_rel(root, varno, NULL);
+
+ /*
+ * If query allows parallelism in general, check whether the
+ * quals are parallel-restricted. (We need not check
+ * final_rel->reltarget because it's empty at this point.
+ * Anything parallel-restricted in the query tlist will be
+ * dealt with later.) This is normally pretty silly, because
+ * a Result-only plan would never be interesting to
+ * parallelize. However, if force_parallel_mode is on, then
+ * we want to execute the Result in a parallel worker if
+ * possible, so we must do this.
+ */
+ if (root->glob->parallelModeOK &&
+ force_parallel_mode != FORCE_PARALLEL_OFF)
+ final_rel->consider_parallel =
+ is_parallel_safe(root, parse->jointree->quals);
+
+ /*
+ * The only path for it is a trivial Result path. We cheat a
+ * bit here by using a GroupResultPath, because that way we
+ * can just jam the quals into it without preprocessing them.
+ * (But, if you hold your head at the right angle, a FROM-less
+ * SELECT is a kind of degenerate-grouping case, so it's not
+ * that much of a cheat.)
+ */
+ add_path(final_rel, (Path *)
+ create_group_result_path(root, final_rel,
+ final_rel->reltarget,
+ (List *) parse->jointree->quals));
+
+ /* Select cheapest path (pretty easy in this case...) */
+ set_cheapest(final_rel);
+
+ /*
+ * We still are required to call qp_callback, in case it's
+ * something like "SELECT 2+2 ORDER BY 1".
+ */
+ (*qp_callback) (root, qp_extra);
+
+ return final_rel;
+ }
+ }
+ }
+
+ /*
* Populate append_rel_array with each AppendRelInfo to allow direct
* lookups by child relid.
*/
diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c
index c729a99..ecc74a7 100644
--- a/src/backend/optimizer/plan/planner.c
+++ b/src/backend/optimizer/plan/planner.c
@@ -606,6 +606,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
List *newWithCheckOptions;
List *newHaving;
bool hasOuterJoins;
+ bool hasResultRTEs;
RelOptInfo *final_rel;
ListCell *l;
@@ -647,6 +648,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
SS_process_ctes(root);
/*
+ * If the FROM clause is empty, replace it with a dummy RTE_RESULT RTE, so
+ * that we don't need so many special cases to deal with that situation.
+ */
+ replace_empty_jointree(parse);
+
+ /*
* Look for ANY and EXISTS SubLinks in WHERE and JOIN/ON clauses, and try
* to transform them into joins. Note that this step does not descend
* into subqueries; if we pull up any subqueries below, their SubLinks are
@@ -679,14 +686,16 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
/*
* Detect whether any rangetable entries are RTE_JOIN kind; if not, we can
- * avoid the expense of doing flatten_join_alias_vars(). Also check for
- * outer joins --- if none, we can skip reduce_outer_joins(). And check
- * for LATERAL RTEs, too. This must be done after we have done
- * pull_up_subqueries(), of course.
+ * avoid the expense of doing flatten_join_alias_vars(). Likewise check
+ * whether any are RTE_RESULT kind; if not, we can skip
+ * remove_useless_result_rtes(). Also check for outer joins --- if none,
+ * we can skip reduce_outer_joins(). And check for LATERAL RTEs, too.
+ * This must be done after we have done pull_up_subqueries(), of course.
*/
root->hasJoinRTEs = false;
root->hasLateralRTEs = false;
hasOuterJoins = false;
+ hasResultRTEs = false;
foreach(l, parse->rtable)
{
RangeTblEntry *rte = lfirst_node(RangeTblEntry, l);
@@ -697,6 +706,10 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
if (IS_OUTER_JOIN(rte->jointype))
hasOuterJoins = true;
}
+ else if (rte->rtekind == RTE_RESULT)
+ {
+ hasResultRTEs = true;
+ }
if (rte->lateral)
root->hasLateralRTEs = true;
}
@@ -712,10 +725,10 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
/*
* Expand any rangetable entries that are inheritance sets into "append
* relations". This can add entries to the rangetable, but they must be
- * plain base relations not joins, so it's OK (and marginally more
- * efficient) to do it after checking for join RTEs. We must do it after
- * pulling up subqueries, else we'd fail to handle inherited tables in
- * subqueries.
+ * plain RTE_RELATION entries, so it's OK (and marginally more efficient)
+ * to do it after checking for joins and other special RTEs. We must do
+ * this after pulling up subqueries, else we'd fail to handle inherited
+ * tables in subqueries.
*/
expand_inherited_tables(root);
@@ -963,6 +976,14 @@ subquery_planner(PlannerGlobal *glob, Query *parse,
reduce_outer_joins(root);
/*
+ * If we have any RTE_RESULT relations, see if they can be deleted from
+ * the jointree. This step is most effectively done after we've done
+ * expression preprocessing and outer join reduction.
+ */
+ if (hasResultRTEs)
+ remove_useless_result_rtes(root);
+
+ /*
* Do the main planning. If we have an inherited target relation, that
* needs special processing, else go straight to grouping_planner.
*/
@@ -3889,9 +3910,9 @@ create_degenerate_grouping_paths(PlannerInfo *root, RelOptInfo *input_rel,
while (--nrows >= 0)
{
path = (Path *)
- create_result_path(root, grouped_rel,
- grouped_rel->reltarget,
- (List *) parse->havingQual);
+ create_group_result_path(root, grouped_rel,
+ grouped_rel->reltarget,
+ (List *) parse->havingQual);
paths = lappend(paths, path);
}
path = (Path *)
@@ -3909,9 +3930,9 @@ create_degenerate_grouping_paths(PlannerInfo *root, RelOptInfo *input_rel,
{
/* No grouping sets, or just one, so one output row */
path = (Path *)
- create_result_path(root, grouped_rel,
- grouped_rel->reltarget,
- (List *) parse->havingQual);
+ create_group_result_path(root, grouped_rel,
+ grouped_rel->reltarget,
+ (List *) parse->havingQual);
}
add_path(grouped_rel, path);
diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c
index 83008d7..39b70cf 100644
--- a/src/backend/optimizer/plan/subselect.c
+++ b/src/backend/optimizer/plan/subselect.c
@@ -1460,12 +1460,6 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink,
return NULL;
/*
- * The subquery must have a nonempty jointree, else we won't have a join.
- */
- if (subselect->jointree->fromlist == NIL)
- return NULL;
-
- /*
* Separate out the WHERE clause. (We could theoretically also remove
* top-level plain JOIN/ON clauses, but it's probably not worth the
* trouble.)
@@ -1494,6 +1488,11 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink,
return NULL;
/*
+ * The subquery must have a nonempty jointree, but we can make it so.
+ */
+ replace_empty_jointree(subselect);
+
+ /*
* Prepare to pull up the sub-select into top range table.
*
* We rely here on the assumption that the outer query has no references
diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c
index cd6e119..780b562 100644
--- a/src/backend/optimizer/prep/prepjointree.c
+++ b/src/backend/optimizer/prep/prepjointree.c
@@ -4,12 +4,14 @@
* Planner preprocessing for subqueries and join tree manipulation.
*
* NOTE: the intended sequence for invoking these operations is
+ * replace_empty_jointree
* pull_up_sublinks
* inline_set_returning_functions
* pull_up_subqueries
* flatten_simple_union_all
* do expression preprocessing (including flattening JOIN alias vars)
* reduce_outer_joins
+ * remove_useless_result_rtes
*
*
* Portions Copyright (c) 1996-2018, PostgreSQL Global Development Group
@@ -66,14 +68,12 @@ static Node *pull_up_sublinks_qual_recurse(PlannerInfo *root, Node *node,
static Node *pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok);
+ AppendRelInfo *containing_appendrel);
static Node *pull_up_simple_subquery(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok);
+ AppendRelInfo *containing_appendrel);
static Node *pull_up_simple_union_all(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte);
static void pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root,
@@ -82,12 +82,10 @@ static void pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root,
static void make_setop_translation_list(Query *query, Index newvarno,
List **translated_vars);
static bool is_simple_subquery(Query *subquery, RangeTblEntry *rte,
- JoinExpr *lowest_outer_join,
- bool deletion_ok);
+ JoinExpr *lowest_outer_join);
static Node *pull_up_simple_values(PlannerInfo *root, Node *jtnode,
RangeTblEntry *rte);
-static bool is_simple_values(PlannerInfo *root, RangeTblEntry *rte,
- bool deletion_ok);
+static bool is_simple_values(PlannerInfo *root, RangeTblEntry *rte);
static bool is_simple_union_all(Query *subquery);
static bool is_simple_union_all_recurse(Node *setOp, Query *setOpQuery,
List *colTypes);
@@ -103,7 +101,6 @@ static Node *pullup_replace_vars_callback(Var *var,
replace_rte_variables_context *context);
static Query *pullup_replace_vars_subquery(Query *query,
pullup_replace_vars_context *context);
-static Node *pull_up_subqueries_cleanup(Node *jtnode);
static reduce_outer_joins_state *reduce_outer_joins_pass1(Node *jtnode);
static void reduce_outer_joins_pass2(Node *jtnode,
reduce_outer_joins_state *state,
@@ -111,14 +108,62 @@ static void reduce_outer_joins_pass2(Node *jtnode,
Relids nonnullable_rels,
List *nonnullable_vars,
List *forced_null_vars);
-static void substitute_multiple_relids(Node *node,
- int varno, Relids subrelids);
+static Node *remove_useless_results_recurse(PlannerInfo *root, Node *jtnode);
+static int is_result_ref(PlannerInfo *root, Node *jtnode);
+static void remove_result_refs(PlannerInfo *root, int varno, Node *newjtloc);
+static bool find_dependent_phvs(Node *node, int varno);
+static void substitute_phv_relids(Node *node,
+ int varno, Relids subrelids);
static void fix_append_rel_relids(List *append_rel_list, int varno,
Relids subrelids);
static Node *find_jointree_node_for_rel(Node *jtnode, int relid);
/*
+ * replace_empty_jointree
+ * If the Query's jointree is empty, replace it with a dummy RTE_RESULT
+ * relation.
+ *
+ * By doing this, we can avoid a bunch of corner cases that formerly existed
+ * for SELECTs with omitted FROM clauses. An example is that a subquery
+ * with empty jointree previously could not be pulled up, because that would
+ * have resulted in an empty relid set, making the subquery not uniquely
+ * identifiable for join or PlaceHolderVar processing.
+ *
+ * Unlike most other functions in this file, this function doesn't recurse;
+ * we rely on other processing to invoke it on sub-queries at suitable times.
+ */
+void
+replace_empty_jointree(Query *parse)
+{
+ RangeTblEntry *rte;
+ Index rti;
+ RangeTblRef *rtr;
+
+ /* Nothing to do if jointree is already nonempty */
+ if (parse->jointree->fromlist != NIL)
+ return;
+
+ /* We mustn't change it in the top level of a setop tree, either */
+ if (parse->setOperations)
+ return;
+
+ /* Create suitable RTE */
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RESULT;
+ rte->eref = makeAlias("*RESULT*", NIL);
+
+ /* Add it to rangetable */
+ parse->rtable = lappend(parse->rtable, rte);
+ rti = list_length(parse->rtable);
+
+ /* And jam a reference into the jointree */
+ rtr = makeNode(RangeTblRef);
+ rtr->rtindex = rti;
+ parse->jointree->fromlist = list_make1(rtr);
+}
+
+/*
* pull_up_sublinks
* Attempt to pull up ANY and EXISTS SubLinks to be treated as
* semijoins or anti-semijoins.
@@ -611,16 +656,11 @@ pull_up_subqueries(PlannerInfo *root)
{
/* Top level of jointree must always be a FromExpr */
Assert(IsA(root->parse->jointree, FromExpr));
- /* Reset flag saying we need a deletion cleanup pass */
- root->hasDeletedRTEs = false;
/* Recursion starts with no containing join nor appendrel */
root->parse->jointree = (FromExpr *)
pull_up_subqueries_recurse(root, (Node *) root->parse->jointree,
- NULL, NULL, NULL, false);
- /* Apply cleanup phase if necessary */
- if (root->hasDeletedRTEs)
- root->parse->jointree = (FromExpr *)
- pull_up_subqueries_cleanup((Node *) root->parse->jointree);
+ NULL, NULL, NULL);
+ /* We should still have a FromExpr */
Assert(IsA(root->parse->jointree, FromExpr));
}
@@ -629,8 +669,6 @@ pull_up_subqueries(PlannerInfo *root)
* Recursive guts of pull_up_subqueries.
*
* This recursively processes the jointree and returns a modified jointree.
- * Or, if it's valid to drop the current node from the jointree completely,
- * it returns NULL.
*
* If this jointree node is within either side of an outer join, then
* lowest_outer_join references the lowest such JoinExpr node; otherwise
@@ -647,37 +685,27 @@ pull_up_subqueries(PlannerInfo *root)
* This forces use of the PlaceHolderVar mechanism for all non-Var targetlist
* items, and puts some additional restrictions on what can be pulled up.
*
- * deletion_ok is true if the caller can cope with us returning NULL for a
- * deletable leaf node (for example, a VALUES RTE that could be pulled up).
- * If it's false, we'll avoid pullup in such cases.
- *
* A tricky aspect of this code is that if we pull up a subquery we have
* to replace Vars that reference the subquery's outputs throughout the
* parent query, including quals attached to jointree nodes above the one
- * we are currently processing! We handle this by being careful not to
- * change the jointree structure while recursing: no nodes other than leaf
- * RangeTblRef entries and entirely-empty FromExprs will be replaced or
- * deleted. Also, we can't turn pullup_replace_vars loose on the whole
- * jointree, because it'll return a mutated copy of the tree; we have to
+ * we are currently processing! We handle this by being careful to maintain
+ * validity of the jointree structure while recursing, in the following sense:
+ * whenever we recurse, all qual expressions in the tree must be reachable
+ * from the top level, in case the recursive call needs to modify them.
+ *
+ * Notice also that we can't turn pullup_replace_vars loose on the whole
+ * jointree, because it'd return a mutated copy of the tree; we have to
* invoke it just on the quals, instead. This behavior is what makes it
* reasonable to pass lowest_outer_join and lowest_nulling_outer_join as
* pointers rather than some more-indirect way of identifying the lowest
* OJs. Likewise, we don't replace append_rel_list members but only their
* substructure, so the containing_appendrel reference is safe to use.
- *
- * Because of the rule that no jointree nodes with substructure can be
- * replaced, we cannot fully handle the case of deleting nodes from the tree:
- * when we delete one child of a JoinExpr, we need to replace the JoinExpr
- * with a FromExpr, and that can't happen here. Instead, we set the
- * root->hasDeletedRTEs flag, which tells pull_up_subqueries() that an
- * additional pass over the tree is needed to clean up.
*/
static Node *
pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok)
+ AppendRelInfo *containing_appendrel)
{
Assert(jtnode != NULL);
if (IsA(jtnode, RangeTblRef))
@@ -693,15 +721,13 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
* unless is_safe_append_member says so.
*/
if (rte->rtekind == RTE_SUBQUERY &&
- is_simple_subquery(rte->subquery, rte,
- lowest_outer_join, deletion_ok) &&
+ is_simple_subquery(rte->subquery, rte, lowest_outer_join) &&
(containing_appendrel == NULL ||
is_safe_append_member(rte->subquery)))
return pull_up_simple_subquery(root, jtnode, rte,
lowest_outer_join,
lowest_nulling_outer_join,
- containing_appendrel,
- deletion_ok);
+ containing_appendrel);
/*
* Alternatively, is it a simple UNION ALL subquery? If so, flatten
@@ -725,7 +751,7 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
if (rte->rtekind == RTE_VALUES &&
lowest_outer_join == NULL &&
containing_appendrel == NULL &&
- is_simple_values(root, rte, deletion_ok))
+ is_simple_values(root, rte))
return pull_up_simple_values(root, jtnode, rte);
/* Otherwise, do nothing at this node. */
@@ -733,50 +759,16 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
else if (IsA(jtnode, FromExpr))
{
FromExpr *f = (FromExpr *) jtnode;
- bool have_undeleted_child = false;
ListCell *l;
Assert(containing_appendrel == NULL);
-
- /*
- * If the FromExpr has quals, it's not deletable even if its parent
- * would allow deletion.
- */
- if (f->quals)
- deletion_ok = false;
-
+ /* Recursively transform all the child nodes */
foreach(l, f->fromlist)
{
- /*
- * In a non-deletable FromExpr, we can allow deletion of child
- * nodes so long as at least one child remains; so it's okay
- * either if any previous child survives, or if there's more to
- * come. If all children are deletable in themselves, we'll force
- * the last one to remain unflattened.
- *
- * As a separate matter, we can allow deletion of all children of
- * the top-level FromExpr in a query, since that's a special case
- * anyway.
- */
- bool sub_deletion_ok = (deletion_ok ||
- have_undeleted_child ||
- lnext(l) != NULL ||
- f == root->parse->jointree);
-
lfirst(l) = pull_up_subqueries_recurse(root, lfirst(l),
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- sub_deletion_ok);
- if (lfirst(l) != NULL)
- have_undeleted_child = true;
- }
-
- if (deletion_ok && !have_undeleted_child)
- {
- /* OK to delete this FromExpr entirely */
- root->hasDeletedRTEs = true; /* probably is set already */
- return NULL;
+ NULL);
}
}
else if (IsA(jtnode, JoinExpr))
@@ -788,22 +780,14 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
switch (j->jointype)
{
case JOIN_INNER:
-
- /*
- * INNER JOIN can allow deletion of either child node, but not
- * both. So right child gets permission to delete only if
- * left child didn't get removed.
- */
j->larg = pull_up_subqueries_recurse(root, j->larg,
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- true);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
lowest_outer_join,
lowest_nulling_outer_join,
- NULL,
- j->larg != NULL);
+ NULL);
break;
case JOIN_LEFT:
case JOIN_SEMI:
@@ -811,37 +795,31 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
lowest_nulling_outer_join,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
j,
- NULL,
- false);
+ NULL);
break;
case JOIN_FULL:
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
j,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
j,
- NULL,
- false);
+ NULL);
break;
case JOIN_RIGHT:
j->larg = pull_up_subqueries_recurse(root, j->larg,
j,
j,
- NULL,
- false);
+ NULL);
j->rarg = pull_up_subqueries_recurse(root, j->rarg,
j,
lowest_nulling_outer_join,
- NULL,
- false);
+ NULL);
break;
default:
elog(ERROR, "unrecognized join type: %d",
@@ -861,8 +839,8 @@ pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode,
*
* jtnode is a RangeTblRef that has been tentatively identified as a simple
* subquery by pull_up_subqueries. We return the replacement jointree node,
- * or NULL if the subquery can be deleted entirely, or jtnode itself if we
- * determine that the subquery can't be pulled up after all.
+ * or jtnode itself if we determine that the subquery can't be pulled up
+ * after all.
*
* rte is the RangeTblEntry referenced by jtnode. Remaining parameters are
* as for pull_up_subqueries_recurse.
@@ -871,8 +849,7 @@ static Node *
pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
JoinExpr *lowest_outer_join,
JoinExpr *lowest_nulling_outer_join,
- AppendRelInfo *containing_appendrel,
- bool deletion_ok)
+ AppendRelInfo *containing_appendrel)
{
Query *parse = root->parse;
int varno = ((RangeTblRef *) jtnode)->rtindex;
@@ -926,6 +903,12 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
Assert(subquery->cteList == NIL);
/*
+ * If the FROM clause is empty, replace it with a dummy RTE_RESULT RTE, so
+ * that we don't need so many special cases to deal with that situation.
+ */
+ replace_empty_jointree(subquery);
+
+ /*
* Pull up any SubLinks within the subquery's quals, so that we don't
* leave unoptimized SubLinks behind.
*/
@@ -957,8 +940,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
* easier just to keep this "if" looking the same as the one in
* pull_up_subqueries_recurse.
*/
- if (is_simple_subquery(subquery, rte,
- lowest_outer_join, deletion_ok) &&
+ if (is_simple_subquery(subquery, rte, lowest_outer_join) &&
(containing_appendrel == NULL || is_safe_append_member(subquery)))
{
/* good to go */
@@ -1159,6 +1141,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
case RTE_JOIN:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* these can't contain any lateral references */
break;
}
@@ -1195,7 +1178,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
Relids subrelids;
subrelids = get_relids_in_jointree((Node *) subquery->jointree, false);
- substitute_multiple_relids((Node *) parse, varno, subrelids);
+ substitute_phv_relids((Node *) parse, varno, subrelids);
fix_append_rel_relids(root->append_rel_list, varno, subrelids);
}
@@ -1235,17 +1218,14 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte,
/*
* Return the adjusted subquery jointree to replace the RangeTblRef entry
- * in parent's jointree; or, if we're flattening a subquery with empty
- * FROM list, return NULL to signal deletion of the subquery from the
- * parent jointree (and set hasDeletedRTEs to ensure cleanup later).
+ * in parent's jointree; or, if the FromExpr is degenerate, just return
+ * its single member.
*/
- if (subquery->jointree->fromlist == NIL)
- {
- Assert(deletion_ok);
- Assert(subquery->jointree->quals == NULL);
- root->hasDeletedRTEs = true;
- return NULL;
- }
+ Assert(IsA(subquery->jointree, FromExpr));
+ Assert(subquery->jointree->fromlist != NIL);
+ if (subquery->jointree->quals == NULL &&
+ list_length(subquery->jointree->fromlist) == 1)
+ return (Node *) linitial(subquery->jointree->fromlist);
return (Node *) subquery->jointree;
}
@@ -1381,7 +1361,7 @@ pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root, int parentRTindex,
rtr = makeNode(RangeTblRef);
rtr->rtindex = childRTindex;
(void) pull_up_subqueries_recurse(root, (Node *) rtr,
- NULL, NULL, appinfo, false);
+ NULL, NULL, appinfo);
}
else if (IsA(setOp, SetOperationStmt))
{
@@ -1436,12 +1416,10 @@ make_setop_translation_list(Query *query, Index newvarno,
* (Note subquery is not necessarily equal to rte->subquery; it could be a
* processed copy of that.)
* lowest_outer_join is the lowest outer join above the subquery, or NULL.
- * deletion_ok is true if it'd be okay to delete the subquery entirely.
*/
static bool
is_simple_subquery(Query *subquery, RangeTblEntry *rte,
- JoinExpr *lowest_outer_join,
- bool deletion_ok)
+ JoinExpr *lowest_outer_join)
{
/*
* Let's just make sure it's a valid subselect ...
@@ -1491,44 +1469,6 @@ is_simple_subquery(Query *subquery, RangeTblEntry *rte,
return false;
/*
- * Don't pull up a subquery with an empty jointree, unless it has no quals
- * and deletion_ok is true and we're not underneath an outer join.
- *
- * query_planner() will correctly generate a Result plan for a jointree
- * that's totally empty, but we can't cope with an empty FromExpr
- * appearing lower down in a jointree: we identify join rels via baserelid
- * sets, so we couldn't distinguish a join containing such a FromExpr from
- * one without it. We can only handle such cases if the place where the
- * subquery is linked is a FromExpr or inner JOIN that would still be
- * nonempty after removal of the subquery, so that it's still identifiable
- * via its contained baserelids. Safe contexts are signaled by
- * deletion_ok.
- *
- * But even in a safe context, we must keep the subquery if it has any
- * quals, because it's unclear where to put them in the upper query.
- *
- * Also, we must forbid pullup if such a subquery is underneath an outer
- * join, because then we might need to wrap its output columns with
- * PlaceHolderVars, and the PHVs would then have empty relid sets meaning
- * we couldn't tell where to evaluate them. (This test is separate from
- * the deletion_ok flag for possible future expansion: deletion_ok tells
- * whether the immediate parent site in the jointree could cope, not
- * whether we'd have PHV issues. It's possible this restriction could be
- * fixed by letting the PHVs use the relids of the parent jointree item,
- * but that complication is for another day.)
- *
- * Note that deletion of a subquery is also dependent on the check below
- * that its targetlist contains no set-returning functions. Deletion from
- * a FROM list or inner JOIN is okay only if the subquery must return
- * exactly one row.
- */
- if (subquery->jointree->fromlist == NIL &&
- (subquery->jointree->quals != NULL ||
- !deletion_ok ||
- lowest_outer_join != NULL))
- return false;
-
- /*
* If the subquery is LATERAL, check for pullup restrictions from that.
*/
if (rte->lateral)
@@ -1602,9 +1542,10 @@ is_simple_subquery(Query *subquery, RangeTblEntry *rte,
* Pull up a single simple VALUES RTE.
*
* jtnode is a RangeTblRef that has been identified as a simple VALUES RTE
- * by pull_up_subqueries. We always return NULL indicating that the RTE
- * can be deleted entirely (all failure cases should have been detected by
- * is_simple_values()).
+ * by pull_up_subqueries. We always return a RangeTblRef representing a
+ * RESULT RTE to replace it (all failure cases should have been detected by
+ * is_simple_values()). Actually, what we return is just jtnode, because
+ * we replace the VALUES RTE in the rangetable with the RESULT RTE.
*
* rte is the RangeTblEntry referenced by jtnode. Because of the limited
* possible usage of VALUES RTEs, we do not need the remaining parameters
@@ -1703,11 +1644,23 @@ pull_up_simple_values(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte)
Assert(root->placeholder_list == NIL);
/*
- * Return NULL to signal deletion of the VALUES RTE from the parent
- * jointree (and set hasDeletedRTEs to ensure cleanup later).
+ * Replace the VALUES RTE with a RESULT RTE. The VALUES RTE is the only
+ * rtable entry in the current query level, so this is easy.
*/
- root->hasDeletedRTEs = true;
- return NULL;
+ Assert(list_length(parse->rtable) == 1);
+
+ /* Create suitable RTE */
+ rte = makeNode(RangeTblEntry);
+ rte->rtekind = RTE_RESULT;
+ rte->eref = makeAlias("*RESULT*", NIL);
+
+ /* Replace rangetable */
+ parse->rtable = list_make1(rte);
+
+ /* We could manufacture a new RangeTblRef, but the one we have is fine */
+ Assert(varno == 1);
+
+ return jtnode;
}
/*
@@ -1716,24 +1669,16 @@ pull_up_simple_values(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte)
* to pull up into the parent query.
*
* rte is the RTE_VALUES RangeTblEntry to check.
- * deletion_ok is true if it'd be okay to delete the VALUES RTE entirely.
*/
static bool
-is_simple_values(PlannerInfo *root, RangeTblEntry *rte, bool deletion_ok)
+is_simple_values(PlannerInfo *root, RangeTblEntry *rte)
{
Assert(rte->rtekind == RTE_VALUES);
/*
- * We can only pull up a VALUES RTE if deletion_ok is true. It's
- * basically the same case as a sub-select with empty FROM list; see
- * comments in is_simple_subquery().
- */
- if (!deletion_ok)
- return false;
-
- /*
- * Also, there must be exactly one VALUES list, else it's not semantically
- * correct to delete the VALUES RTE.
+ * There must be exactly one VALUES list, else it's not semantically
+ * correct to replace the VALUES RTE with a RESULT RTE, nor would we have
+ * a unique set of expressions to substitute into the parent query.
*/
if (list_length(rte->values_lists) != 1)
return false;
@@ -1746,8 +1691,8 @@ is_simple_values(PlannerInfo *root, RangeTblEntry *rte, bool deletion_ok)
/*
* Don't pull up a VALUES that contains any set-returning or volatile
- * functions. Again, the considerations here are basically identical to
- * restrictions on a subquery's targetlist.
+ * functions. The considerations here are basically identical to the
+ * restrictions on a pull-able subquery's targetlist.
*/
if (expression_returns_set((Node *) rte->values_lists) ||
contain_volatile_functions((Node *) rte->values_lists))
@@ -1850,7 +1795,9 @@ is_safe_append_member(Query *subquery)
/*
* It's only safe to pull up the child if its jointree contains exactly
* one RTE, else the AppendRelInfo data structure breaks. The one base RTE
- * could be buried in several levels of FromExpr, however.
+ * could be buried in several levels of FromExpr, however. Also, if the
+ * child's jointree is completely empty, we can pull up because
+ * pull_up_simple_subquery will insert a single RTE_RESULT RTE instead.
*
* Also, the child can't have any WHERE quals because there's no place to
* put them in an appendrel. (This is a bit annoying...) If we didn't
@@ -1859,6 +1806,11 @@ is_safe_append_member(Query *subquery)
* fix_append_rel_relids().
*/
jtnode = subquery->jointree;
+ Assert(IsA(jtnode, FromExpr));
+ /* Check the completely-empty case */
+ if (jtnode->fromlist == NIL && jtnode->quals == NULL)
+ return true;
+ /* Check the more general case */
while (IsA(jtnode, FromExpr))
{
if (jtnode->quals != NULL)
@@ -2014,6 +1966,7 @@ replace_vars_in_jointree(Node *jtnode,
case RTE_JOIN:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* these shouldn't be marked LATERAL */
Assert(false);
break;
@@ -2290,65 +2243,6 @@ pullup_replace_vars_subquery(Query *query,
NULL);
}
-/*
- * pull_up_subqueries_cleanup
- * Recursively fix up jointree after deletion of some subqueries.
- *
- * The jointree now contains some NULL subtrees, which we need to get rid of.
- * In a FromExpr, just rebuild the child-node list with null entries deleted.
- * In an inner JOIN, replace the JoinExpr node with a one-child FromExpr.
- */
-static Node *
-pull_up_subqueries_cleanup(Node *jtnode)
-{
- Assert(jtnode != NULL);
- if (IsA(jtnode, RangeTblRef))
- {
- /* Nothing to do at leaf nodes. */
- }
- else if (IsA(jtnode, FromExpr))
- {
- FromExpr *f = (FromExpr *) jtnode;
- List *newfrom = NIL;
- ListCell *l;
-
- foreach(l, f->fromlist)
- {
- Node *child = (Node *) lfirst(l);
-
- if (child == NULL)
- continue;
- child = pull_up_subqueries_cleanup(child);
- newfrom = lappend(newfrom, child);
- }
- f->fromlist = newfrom;
- }
- else if (IsA(jtnode, JoinExpr))
- {
- JoinExpr *j = (JoinExpr *) jtnode;
-
- if (j->larg)
- j->larg = pull_up_subqueries_cleanup(j->larg);
- if (j->rarg)
- j->rarg = pull_up_subqueries_cleanup(j->rarg);
- if (j->larg == NULL)
- {
- Assert(j->jointype == JOIN_INNER);
- Assert(j->rarg != NULL);
- return (Node *) makeFromExpr(list_make1(j->rarg), j->quals);
- }
- else if (j->rarg == NULL)
- {
- Assert(j->jointype == JOIN_INNER);
- return (Node *) makeFromExpr(list_make1(j->larg), j->quals);
- }
- }
- else
- elog(ERROR, "unrecognized node type: %d",
- (int) nodeTag(jtnode));
- return jtnode;
-}
-
/*
* flatten_simple_union_all
@@ -2858,9 +2752,405 @@ reduce_outer_joins_pass2(Node *jtnode,
(int) nodeTag(jtnode));
}
+
+/*
+ * remove_useless_result_rtes
+ * Attempt to remove RTE_RESULT RTEs from the join tree.
+ *
+ * We can remove RTE_RESULT entries from the join tree using the knowledge
+ * that RTE_RESULT returns exactly one row and has no output columns. Hence,
+ * if one is inner-joined to anything else, we can delete it. Optimizations
+ * are also possible for some outer-join cases, as detailed below.
+ *
+ * Some of these optimizations depend on recognizing empty (constant-true)
+ * quals for FromExprs and JoinExprs. That makes it useful to apply this
+ * optimization pass after expression preprocessing, since that will have
+ * eliminated constant-true quals, allowing more cases to be recognized as
+ * optimizable. What's more, the usual reason for an RTE_RESULT to be present
+ * is that we pulled up a subquery or VALUES clause, thus very possibly
+ * replacing Vars with constants, making it more likely that a qual can be
+ * reduced to constant true. Also, because some optimizations depend on
+ * the outer-join type, it's best to have done reduce_outer_joins() first.
+ *
+ * A PlaceHolderVar referencing an RTE_RESULT RTE poses an obstacle to this
+ * process: we must remove the RTE_RESULT's relid from the PHV's phrels, but
+ * we must not reduce the phrels set to empty. If that would happen, and
+ * the RTE_RESULT is an immediate child of an outer join, we have to give up
+ * and not remove the RTE_RESULT: there is noplace else to evaluate the
+ * PlaceHolderVar. (That is, in such cases the RTE_RESULT *does* have output
+ * columns.) But if the RTE_RESULT is an immediate child of an inner join,
+ * we can change the PlaceHolderVar's phrels so as to evaluate it at the
+ * inner join instead. This is OK because we really only care that PHVs are
+ * evaluated above or below the correct outer joins.
+ *
+ * We used to try to do this work as part of pull_up_subqueries() where the
+ * potentially-optimizable cases get introduced; but it's way simpler, and
+ * more effective, to do it separately.
+ */
+void
+remove_useless_result_rtes(PlannerInfo *root)
+{
+ ListCell *cell;
+ ListCell *prev;
+ ListCell *next;
+
+ /* Top level of jointree must always be a FromExpr */
+ Assert(IsA(root->parse->jointree, FromExpr));
+ /* Recurse ... */
+ root->parse->jointree = (FromExpr *)
+ remove_useless_results_recurse(root, (Node *) root->parse->jointree);
+ /* We should still have a FromExpr */
+ Assert(IsA(root->parse->jointree, FromExpr));
+
+ /*
+ * Remove any PlanRowMark referencing an RTE_RESULT RTE. We obviously
+ * must do that for any RTE_RESULT that we just removed. But one for a
+ * RTE that we did not remove can be dropped anyway: since the RTE has
+ * only one possible output row, there is no need for EPQ to mark and
+ * restore that row.
+ *
+ * Actually, this concern is moot anyway, because PlanRowMarks cannot be
+ * attached to RTEs that are underneath an outer join, and we'll have
+ * removed every RTE_RESULT that is not underneath an outer join, except
+ * in the case where the only RTE left in the jointree is a RTE_RESULT, ie
+ * something like "SELECT * FROM (SELECT 1 AS x) ss FOR UPDATE". And
+ * obviously EPQ will never fire for that query.
+ *
+ * The reason for being tense about this is mainly that if we let such a
+ * PlanRowMark survive, we'll generate a whole-row Var for the RTE_RESULT,
+ * which the executor will fail to cope with. Rather than add useless
+ * runtime logic to handle that, let's be sure to remove the PlanRowMark.
+ */
+ prev = NULL;
+ for (cell = list_head(root->rowMarks); cell; cell = next)
+ {
+ PlanRowMark *rc = (PlanRowMark *) lfirst(cell);
+
+ next = lnext(cell);
+ if (rt_fetch(rc->rti, root->parse->rtable)->rtekind == RTE_RESULT)
+ root->rowMarks = list_delete_cell(root->rowMarks, cell, prev);
+ else
+ prev = cell;
+ }
+}
+
+/*
+ * remove_useless_results_recurse
+ * Recursive guts of remove_useless_result_rtes.
+ *
+ * This recursively processes the jointree and returns a modified jointree.
+ */
+static Node *
+remove_useless_results_recurse(PlannerInfo *root, Node *jtnode)
+{
+ Assert(jtnode != NULL);
+ if (IsA(jtnode, RangeTblRef))
+ {
+ /* Can't immediately do anything with a RangeTblRef */
+ }
+ else if (IsA(jtnode, FromExpr))
+ {
+ FromExpr *f = (FromExpr *) jtnode;
+ Relids result_relids = NULL;
+ ListCell *cell;
+ ListCell *prev;
+ ListCell *next;
+
+ /*
+ * We can drop RTE_RESULT rels from the fromlist so long as at least
+ * one child remains, since joining to a one-row table changes
+ * nothing. The easiest way to mechanize this rule is to modify the
+ * list in-place, using list_delete_cell.
+ */
+ prev = NULL;
+ for (cell = list_head(f->fromlist); cell; cell = next)
+ {
+ Node *child = (Node *) lfirst(cell);
+ int varno;
+
+ /* Recursively transform child ... */
+ child = remove_useless_results_recurse(root, child);
+ /* ... and stick it back into the tree */
+ lfirst(cell) = child;
+ next = lnext(cell);
+
+ /*
+ * If it's an RTE_RESULT with at least one sibling, we can drop
+ * it. We don't yet know what the inner join's final relid set
+ * will be, so postpone cleanup of PHVs etc till after this loop.
+ */
+ if (list_length(f->fromlist) > 1 &&
+ (varno = is_result_ref(root, child)) != 0)
+ {
+ f->fromlist = list_delete_cell(f->fromlist, cell, prev);
+ result_relids = bms_add_member(result_relids, varno);
+ }
+ else
+ prev = cell;
+ }
+
+ /*
+ * Clean up if we dropped any RTE_RESULT RTEs. This is a bit
+ * inefficient if there's more than one, but it seems better to
+ * optimize the support code for the single-relid case.
+ */
+ if (result_relids)
+ {
+ int varno;
+
+ while ((varno = bms_first_member(result_relids)) >= 0)
+ remove_result_refs(root, varno, (Node *) f);
+ }
+
+ /*
+ * If we're not at the top of the jointree, it's valid to simplify a
+ * degenerate FromExpr into its single child. (At the top, we must
+ * keep the FromExpr since Query.jointree is required to point to a
+ * FromExpr.)
+ */
+ if (f != root->parse->jointree &&
+ f->quals == NULL &&
+ list_length(f->fromlist) == 1)
+ return (Node *) linitial(f->fromlist);
+ }
+ else if (IsA(jtnode, JoinExpr))
+ {
+ JoinExpr *j = (JoinExpr *) jtnode;
+ int varno;
+
+ /* First, recurse */
+ j->larg = remove_useless_results_recurse(root, j->larg);
+ j->rarg = remove_useless_results_recurse(root, j->rarg);
+
+ /* Apply join-type-specific optimization rules */
+ switch (j->jointype)
+ {
+ case JOIN_INNER:
+
+ /*
+ * An inner join is equivalent to a FromExpr, so if either
+ * side was simplified to an RTE_RESULT rel, we can replace
+ * the join with a FromExpr with just the other side; and if
+ * the qual is empty (JOIN ON TRUE) then we can omit the
+ * FromExpr as well.
+ */
+ if ((varno = is_result_ref(root, j->larg)) != 0)
+ {
+ remove_result_refs(root, varno, j->rarg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->rarg), j->quals);
+ else
+ jtnode = j->rarg;
+ }
+ else if ((varno = is_result_ref(root, j->rarg)) != 0)
+ {
+ remove_result_refs(root, varno, j->larg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->larg), j->quals);
+ else
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_LEFT:
+
+ /*
+ * We can simplify this case if the RHS is an RTE_RESULT, with
+ * two different possibilities:
+ *
+ * If the qual is empty (JOIN ON TRUE), then the join can be
+ * strength-reduced to a plain inner join, since each LHS row
+ * necessarily has exactly one join partner. So we can always
+ * discard the RHS, much as in the JOIN_INNER case above.
+ *
+ * Otherwise, it's still true that each LHS row should be
+ * returned exactly once, and since the RHS returns no columns
+ * (unless there are PHVs that have to be evaluated there), we
+ * don't much care if it's null-extended or not. So in this
+ * case also, we can just ignore the qual and discard the left
+ * join.
+ */
+ if ((varno = is_result_ref(root, j->rarg)) != 0 &&
+ (j->quals == NULL ||
+ !find_dependent_phvs((Node *) root->parse, varno)))
+ {
+ remove_result_refs(root, varno, j->larg);
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_RIGHT:
+ /* Mirror-image of the JOIN_LEFT case */
+ if ((varno = is_result_ref(root, j->larg)) != 0 &&
+ (j->quals == NULL ||
+ !find_dependent_phvs((Node *) root->parse, varno)))
+ {
+ remove_result_refs(root, varno, j->rarg);
+ jtnode = j->rarg;
+ }
+ break;
+ case JOIN_SEMI:
+
+ /*
+ * We may simplify this case if the RHS is an RTE_RESULT; the
+ * join qual becomes effectively just a filter qual for the
+ * LHS, since we should either return the LHS row or not. For
+ * simplicity we inject the filter qual into a new FromExpr.
+ *
+ * However, we can't simplify if there are PHVs to evaluate at
+ * the RTE_RESULT ... but that's impossible isn't it?
+ */
+ if ((varno = is_result_ref(root, j->rarg)) != 0)
+ {
+ Assert(!find_dependent_phvs((Node *) root->parse, varno));
+ remove_result_refs(root, varno, j->larg);
+ if (j->quals)
+ jtnode = (Node *)
+ makeFromExpr(list_make1(j->larg), j->quals);
+ else
+ jtnode = j->larg;
+ }
+ break;
+ case JOIN_FULL:
+ case JOIN_ANTI:
+ /* We have no special smarts for these cases */
+ break;
+ default:
+ elog(ERROR, "unrecognized join type: %d",
+ (int) j->jointype);
+ break;
+ }
+ }
+ else
+ elog(ERROR, "unrecognized node type: %d",
+ (int) nodeTag(jtnode));
+ return jtnode;
+}
+
+/*
+ * is_result_ref
+ * If jtnode is a RangeTblRef for an RTE_RESULT RTE, return its relid;
+ * otherwise return 0.
+ */
+static inline int
+is_result_ref(PlannerInfo *root, Node *jtnode)
+{
+ int varno;
+
+ if (!IsA(jtnode, RangeTblRef))
+ return 0;
+ varno = ((RangeTblRef *) jtnode)->rtindex;
+ if (rt_fetch(varno, root->parse->rtable)->rtekind != RTE_RESULT)
+ return 0;
+ return varno;
+}
+
+/*
+ * remove_result_refs
+ * Helper routine for dropping an unneeded RTE_RESULT RTE.
+ *
+ * This doesn't physically remove the RTE from the jointree, because that's
+ * more easily handled in remove_useless_results_recurse. What it does do
+ * is the necessary cleanup in the rest of the tree: we must adjust any PHVs
+ * that may reference the RTE. Be sure to call this at a point where the
+ * jointree is valid (no disconnected nodes).
+ *
+ * Note that we don't need to process the append_rel_list, since RTEs
+ * referenced directly in the jointree won't be appendrel members.
+ *
+ * varno is the RTE_RESULT's relid.
+ * newjtloc is the jointree location at which any PHVs referencing the
+ * RTE_RESULT should be evaluated instead.
+ */
+static void
+remove_result_refs(PlannerInfo *root, int varno, Node *newjtloc)
+{
+ /* Fix up PlaceHolderVars as needed */
+ /* If there are no PHVs anywhere, we can skip this bit */
+ if (root->glob->lastPHId != 0)
+ {
+ Relids subrelids;
+
+ subrelids = get_relids_in_jointree(newjtloc, false);
+ Assert(!bms_is_empty(subrelids));
+ substitute_phv_relids((Node *) root->parse, varno, subrelids);
+ }
+
+ /*
+ * We also need to remove any PlanRowMark referencing the RTE, but we
+ * postpone that work until we return to remove_useless_result_rtes.
+ */
+}
+
+
+/*
+ * find_dependent_phvs - are there any PlaceHolderVars whose relids are
+ * exactly the given varno?
+ */
+
+typedef struct
+{
+ Relids relids;
+ int sublevels_up;
+} find_dependent_phvs_context;
+
+static bool
+find_dependent_phvs_walker(Node *node,
+ find_dependent_phvs_context *context)
+{
+ if (node == NULL)
+ return false;
+ if (IsA(node, PlaceHolderVar))
+ {
+ PlaceHolderVar *phv = (PlaceHolderVar *) node;
+
+ if (phv->phlevelsup == context->sublevels_up &&
+ bms_equal(context->relids, phv->phrels))
+ return true;
+ /* fall through to examine children */
+ }
+ if (IsA(node, Query))
+ {
+ /* Recurse into subselects */
+ bool result;
+
+ context->sublevels_up++;
+ result = query_tree_walker((Query *) node,
+ find_dependent_phvs_walker,
+ (void *) context, 0);
+ context->sublevels_up--;
+ return result;
+ }
+ /* Shouldn't need to handle planner auxiliary nodes here */
+ Assert(!IsA(node, SpecialJoinInfo));
+ Assert(!IsA(node, AppendRelInfo));
+ Assert(!IsA(node, PlaceHolderInfo));
+ Assert(!IsA(node, MinMaxAggInfo));
+
+ return expression_tree_walker(node, find_dependent_phvs_walker,
+ (void *) context);
+}
+
+static bool
+find_dependent_phvs(Node *node, int varno)
+{
+ find_dependent_phvs_context context;
+
+ context.relids = bms_make_singleton(varno);
+ context.sublevels_up = 0;
+
+ /*
+ * Must be prepared to start with a Query or a bare expression tree.
+ */
+ return query_or_expression_tree_walker(node,
+ find_dependent_phvs_walker,
+ (void *) &context,
+ 0);
+}
+
/*
- * substitute_multiple_relids - adjust node relid sets after pulling up
- * a subquery
+ * substitute_phv_relids - adjust PlaceHolderVar relid sets after pulling up
+ * a subquery or removing an RTE_RESULT jointree item
*
* Find any PlaceHolderVar nodes in the given tree that reference the
* pulled-up relid, and change them to reference the replacement relid(s).
@@ -2876,11 +3166,11 @@ typedef struct
int varno;
int sublevels_up;
Relids subrelids;
-} substitute_multiple_relids_context;
+} substitute_phv_relids_context;
static bool
-substitute_multiple_relids_walker(Node *node,
- substitute_multiple_relids_context *context)
+substitute_phv_relids_walker(Node *node,
+ substitute_phv_relids_context *context)
{
if (node == NULL)
return false;
@@ -2895,6 +3185,8 @@ substitute_multiple_relids_walker(Node *node,
context->subrelids);
phv->phrels = bms_del_member(phv->phrels,
context->varno);
+ /* Assert we haven't broken the PHV */
+ Assert(!bms_is_empty(phv->phrels));
}
/* fall through to examine children */
}
@@ -2905,7 +3197,7 @@ substitute_multiple_relids_walker(Node *node,
context->sublevels_up++;
result = query_tree_walker((Query *) node,
- substitute_multiple_relids_walker,
+ substitute_phv_relids_walker,
(void *) context, 0);
context->sublevels_up--;
return result;
@@ -2916,14 +3208,14 @@ substitute_multiple_relids_walker(Node *node,
Assert(!IsA(node, PlaceHolderInfo));
Assert(!IsA(node, MinMaxAggInfo));
- return expression_tree_walker(node, substitute_multiple_relids_walker,
+ return expression_tree_walker(node, substitute_phv_relids_walker,
(void *) context);
}
static void
-substitute_multiple_relids(Node *node, int varno, Relids subrelids)
+substitute_phv_relids(Node *node, int varno, Relids subrelids)
{
- substitute_multiple_relids_context context;
+ substitute_phv_relids_context context;
context.varno = varno;
context.sublevels_up = 0;
@@ -2933,7 +3225,7 @@ substitute_multiple_relids(Node *node, int varno, Relids subrelids)
* Must be prepared to start with a Query or a bare expression tree.
*/
query_or_expression_tree_walker(node,
- substitute_multiple_relids_walker,
+ substitute_phv_relids_walker,
(void *) &context,
0);
}
@@ -2943,7 +3235,7 @@ substitute_multiple_relids(Node *node, int varno, Relids subrelids)
*
* When we pull up a subquery, any AppendRelInfo references to the subquery's
* RT index have to be replaced by the substituted relid (and there had better
- * be only one). We also need to apply substitute_multiple_relids to their
+ * be only one). We also need to apply substitute_phv_relids to their
* translated_vars lists, since those might contain PlaceHolderVars.
*
* We assume we may modify the AppendRelInfo nodes in-place.
@@ -2974,9 +3266,9 @@ fix_append_rel_relids(List *append_rel_list, int varno, Relids subrelids)
appinfo->child_relid = subvarno;
}
- /* Also finish fixups for its translated vars */
- substitute_multiple_relids((Node *) appinfo->translated_vars,
- varno, subrelids);
+ /* Also fix up any PHVs in its translated vars */
+ substitute_phv_relids((Node *) appinfo->translated_vars,
+ varno, subrelids);
}
}
diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c
index ee6f4cd..23f9559 100644
--- a/src/backend/optimizer/util/clauses.c
+++ b/src/backend/optimizer/util/clauses.c
@@ -1661,7 +1661,7 @@ contain_leaked_vars_walker(Node *node, void *context)
* find_nonnullable_vars() is that the tested conditions really are different:
* a clause like "t1.v1 IS NOT NULL OR t1.v2 IS NOT NULL" does not prove
* that either v1 or v2 can't be NULL, but it does prove that the t1 row
- * as a whole can't be all-NULL.
+ * as a whole can't be all-NULL. Also, the behavior for PHVs is different.
*
* top_level is true while scanning top-level AND/OR structure; here, showing
* the result is either FALSE or NULL is good enough. top_level is false when
@@ -1847,7 +1847,24 @@ find_nonnullable_rels_walker(Node *node, bool top_level)
{
PlaceHolderVar *phv = (PlaceHolderVar *) node;
+ /*
+ * If the contained expression forces any rels non-nullable, so does
+ * the PHV.
+ */
result = find_nonnullable_rels_walker((Node *) phv->phexpr, top_level);
+
+ /*
+ * If the PHV's syntactic scope is exactly one rel, it will be forced
+ * to be evaluated at that rel, and so it will behave like a Var of
+ * that rel: if the rel's entire output goes to null, so will the PHV.
+ * (If the syntactic scope is a join, we know that the PHV will go to
+ * null if the whole join does; but that is AND semantics while we
+ * need OR semantics for find_nonnullable_rels' result, so we can't do
+ * anything with the knowledge.)
+ */
+ if (phv->phlevelsup == 0 &&
+ bms_membership(phv->phrels) == BMS_SINGLETON)
+ result = bms_add_members(result, phv->phrels);
}
return result;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d50d86b..b48c958 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -1429,17 +1429,17 @@ create_merge_append_path(PlannerInfo *root,
}
/*
- * create_result_path
+ * create_group_result_path
* Creates a path representing a Result-and-nothing-else plan.
*
- * This is only used for degenerate cases, such as a query with an empty
- * jointree.
+ * This is only used for degenerate grouping cases, in which we know we
+ * need to produce one result row, possibly filtered by a HAVING qual.
*/
-ResultPath *
-create_result_path(PlannerInfo *root, RelOptInfo *rel,
- PathTarget *target, List *resconstantqual)
+GroupResultPath *
+create_group_result_path(PlannerInfo *root, RelOptInfo *rel,
+ PathTarget *target, List *havingqual)
{
- ResultPath *pathnode = makeNode(ResultPath);
+ GroupResultPath *pathnode = makeNode(GroupResultPath);
pathnode->path.pathtype = T_Result;
pathnode->path.parent = rel;
@@ -1449,9 +1449,13 @@ create_result_path(PlannerInfo *root, RelOptInfo *rel,
pathnode->path.parallel_safe = rel->consider_parallel;
pathnode->path.parallel_workers = 0;
pathnode->path.pathkeys = NIL;
- pathnode->quals = resconstantqual;
+ pathnode->quals = havingqual;
- /* Hardly worth defining a cost_result() function ... just do it */
+ /*
+ * We can't quite use cost_resultscan() because the quals we want to
+ * account for are not baserestrict quals of the rel. Might as well just
+ * hack it here.
+ */
pathnode->path.rows = 1;
pathnode->path.startup_cost = target->cost.startup;
pathnode->path.total_cost = target->cost.startup +
@@ -1461,12 +1465,12 @@ create_result_path(PlannerInfo *root, RelOptInfo *rel,
* Add cost of qual, if any --- but we ignore its selectivity, since our
* rowcount estimate should be 1 no matter what the qual is.
*/
- if (resconstantqual)
+ if (havingqual)
{
QualCost qual_cost;
- cost_qual_eval(&qual_cost, resconstantqual, root);
- /* resconstantqual is evaluated once at startup */
+ cost_qual_eval(&qual_cost, havingqual, root);
+ /* havingqual is evaluated once at startup */
pathnode->path.startup_cost += qual_cost.startup + qual_cost.per_tuple;
pathnode->path.total_cost += qual_cost.startup + qual_cost.per_tuple;
}
@@ -2020,6 +2024,32 @@ create_namedtuplestorescan_path(PlannerInfo *root, RelOptInfo *rel,
}
/*
+ * create_resultscan_path
+ * Creates a path corresponding to a scan of an RTE_RESULT relation,
+ * returning the pathnode.
+ */
+Path *
+create_resultscan_path(PlannerInfo *root, RelOptInfo *rel,
+ Relids required_outer)
+{
+ Path *pathnode = makeNode(Path);
+
+ pathnode->pathtype = T_Result;
+ pathnode->parent = rel;
+ pathnode->pathtarget = rel->reltarget;
+ pathnode->param_info = get_baserel_parampathinfo(root, rel,
+ required_outer);
+ pathnode->parallel_aware = false;
+ pathnode->parallel_safe = rel->consider_parallel;
+ pathnode->parallel_workers = 0;
+ pathnode->pathkeys = NIL; /* result is always unordered */
+
+ cost_resultscan(pathnode, root, rel, pathnode->param_info);
+
+ return pathnode;
+}
+
+/*
* create_worktablescan_path
* Creates a path corresponding to a scan of a self-reference CTE,
* returning the pathnode.
@@ -3559,6 +3589,11 @@ reparameterize_path(PlannerInfo *root, Path *path,
spath->path.pathkeys,
required_outer);
}
+ case T_Result:
+ /* Supported only for RTE_RESULT scan paths */
+ if (IsA(path, Path))
+ return create_resultscan_path(root, rel, required_outer);
+ break;
case T_Append:
{
AppendPath *apath = (AppendPath *) path;
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 8369e3a..2196189 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -1629,6 +1629,7 @@ build_physical_tlist(PlannerInfo *root, RelOptInfo *rel)
case RTE_VALUES:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* Not all of these can have dropped cols, but share code anyway */
expandRTE(rte, varno, 0, -1, true /* include dropped */ ,
NULL, &colvars);
diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c
index 39f5729..cacf876 100644
--- a/src/backend/optimizer/util/relnode.c
+++ b/src/backend/optimizer/util/relnode.c
@@ -232,10 +232,12 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptInfo *parent)
case RTE_VALUES:
case RTE_CTE:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* Subquery, function, tablefunc, values list, CTE, or ENR --- set
- * up attr range and arrays
+ * up attr range and arrays. RTE_RESULT has no columns, but for
+ * simplicity process it here too.
*
* Note: 0 is included in range to support whole-row Vars
*/
@@ -1108,36 +1110,6 @@ subbuild_joinrel_joinlist(RelOptInfo *joinrel,
/*
- * build_empty_join_rel
- * Build a dummy join relation describing an empty set of base rels.
- *
- * This is used for queries with empty FROM clauses, such as "SELECT 2+2" or
- * "INSERT INTO foo VALUES(...)". We don't try very hard to make the empty
- * joinrel completely valid, since no real planning will be done with it ---
- * we just need it to carry a simple Result path out of query_planner().
- */
-RelOptInfo *
-build_empty_join_rel(PlannerInfo *root)
-{
- RelOptInfo *joinrel;
-
- /* The dummy join relation should be the only one ... */
- Assert(root->join_rel_list == NIL);
-
- joinrel = makeNode(RelOptInfo);
- joinrel->reloptkind = RELOPT_JOINREL;
- joinrel->relids = NULL; /* empty set */
- joinrel->rows = 1; /* we produce one row for such cases */
- joinrel->rtekind = RTE_JOIN;
- joinrel->reltarget = create_empty_pathtarget();
-
- root->join_rel_list = lappend(root->join_rel_list, joinrel);
-
- return joinrel;
-}
-
-
-/*
* fetch_upper_rel
* Build a RelOptInfo describing some post-scan/join query processing,
* or return a pre-existing one if somebody already built it.
diff --git a/src/backend/parser/analyze.c b/src/backend/parser/analyze.c
index 226927b..1d01256 100644
--- a/src/backend/parser/analyze.c
+++ b/src/backend/parser/analyze.c
@@ -2878,6 +2878,9 @@ transformLockingClause(ParseState *pstate, Query *qry, LockingClause *lc,
LCS_asString(lc->strength)),
parser_errposition(pstate, thisrel->location)));
break;
+
+ /* Shouldn't be possible to see RTE_RESULT here */
+
default:
elog(ERROR, "unrecognized RTE type: %d",
(int) rte->rtekind);
diff --git a/src/backend/parser/parse_relation.c b/src/backend/parser/parse_relation.c
index f115ed8..f6413f3 100644
--- a/src/backend/parser/parse_relation.c
+++ b/src/backend/parser/parse_relation.c
@@ -2517,6 +2517,9 @@ expandRTE(RangeTblEntry *rte, int rtindex, int sublevels_up,
}
}
break;
+ case RTE_RESULT:
+ /* These expose no columns, so nothing to do */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
}
@@ -2909,6 +2912,14 @@ get_rte_attribute_type(RangeTblEntry *rte, AttrNumber attnum,
rte->eref->aliasname)));
}
break;
+ case RTE_RESULT:
+ /* this probably can't happen ... */
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column %d of relation \"%s\" does not exist",
+ attnum,
+ rte->eref->aliasname)));
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
}
@@ -3037,6 +3048,15 @@ get_rte_attribute_is_dropped(RangeTblEntry *rte, AttrNumber attnum)
result = false; /* keep compiler quiet */
}
break;
+ case RTE_RESULT:
+ /* this probably can't happen ... */
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_COLUMN),
+ errmsg("column %d of relation \"%s\" does not exist",
+ attnum,
+ rte->eref->aliasname)));
+ result = false; /* keep compiler quiet */
+ break;
default:
elog(ERROR, "unrecognized RTE kind: %d", (int) rte->rtekind);
result = false; /* keep compiler quiet */
diff --git a/src/backend/parser/parse_target.c b/src/backend/parser/parse_target.c
index 4932e58..8801ee0 100644
--- a/src/backend/parser/parse_target.c
+++ b/src/backend/parser/parse_target.c
@@ -398,6 +398,7 @@ markTargetListOrigin(ParseState *pstate, TargetEntry *tle,
case RTE_VALUES:
case RTE_TABLEFUNC:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/* not a simple relation, leave it unmarked */
break;
case RTE_CTE:
@@ -1525,6 +1526,7 @@ expandRecordVariable(ParseState *pstate, Var *var, int levelsup)
case RTE_RELATION:
case RTE_VALUES:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* This case should not occur: a column of a table, values list,
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index f6693ea..d1ca1aa 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -7009,6 +7009,7 @@ get_name_for_var_field(Var *var, int fieldno,
case RTE_RELATION:
case RTE_VALUES:
case RTE_NAMEDTUPLESTORE:
+ case RTE_RESULT:
/*
* This case should not occur: a column of a table, values list,
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index cac6ff0..67a201a 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -237,7 +237,7 @@ typedef enum NodeTag
T_HashPath,
T_AppendPath,
T_MergeAppendPath,
- T_ResultPath,
+ T_GroupResultPath,
T_MaterialPath,
T_UniquePath,
T_GatherPath,
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index aa4a0db..2fdcf67 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -951,7 +951,8 @@ typedef enum RTEKind
RTE_TABLEFUNC, /* TableFunc(.., column list) */
RTE_VALUES, /* VALUES (<exprlist>), (<exprlist>), ... */
RTE_CTE, /* common table expr (WITH list element) */
- RTE_NAMEDTUPLESTORE /* tuplestore, e.g. for AFTER triggers */
+ RTE_NAMEDTUPLESTORE, /* tuplestore, e.g. for AFTER triggers */
+ RTE_RESULT /* RTE represents an omitted FROM clause */
} RTEKind;
typedef struct RangeTblEntry
diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h
index 88d3723..359b044 100644
--- a/src/include/nodes/relation.h
+++ b/src/include/nodes/relation.h
@@ -323,7 +323,6 @@ typedef struct PlannerInfo
* partitioned table */
bool hasJoinRTEs; /* true if any RTEs are RTE_JOIN kind */
bool hasLateralRTEs; /* true if any RTEs are marked LATERAL */
- bool hasDeletedRTEs; /* true if any RTE was deleted from jointree */
bool hasHavingQual; /* true if havingQual was non-null */
bool hasPseudoConstantQuals; /* true if any RestrictInfo has
* pseudoconstant = true */
@@ -1345,17 +1343,17 @@ typedef struct MergeAppendPath
} MergeAppendPath;
/*
- * ResultPath represents use of a Result plan node to compute a variable-free
- * targetlist with no underlying tables (a "SELECT expressions" query).
- * The query could have a WHERE clause, too, represented by "quals".
+ * GroupResultPath represents use of a Result plan node to compute the
+ * output of a degenerate GROUP BY case, wherein we know we should produce
+ * exactly one row, which might then be filtered by a HAVING qual.
*
* Note that quals is a list of bare clauses, not RestrictInfos.
*/
-typedef struct ResultPath
+typedef struct GroupResultPath
{
Path path;
List *quals;
-} ResultPath;
+} GroupResultPath;
/*
* MaterialPath represents use of a Material plan node, i.e., caching of
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 77ca7ff..023733f 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -105,6 +105,8 @@ extern void cost_ctescan(Path *path, PlannerInfo *root,
RelOptInfo *baserel, ParamPathInfo *param_info);
extern void cost_namedtuplestorescan(Path *path, PlannerInfo *root,
RelOptInfo *baserel, ParamPathInfo *param_info);
+extern void cost_resultscan(Path *path, PlannerInfo *root,
+ RelOptInfo *baserel, ParamPathInfo *param_info);
extern void cost_recursive_union(Path *runion, Path *nrterm, Path *rterm);
extern void cost_sort(Path *path, PlannerInfo *root,
List *pathkeys, Cost input_cost, double tuples, int width,
@@ -196,6 +198,7 @@ extern void set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel,
double cte_rows);
extern void set_tablefunc_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern void set_namedtuplestore_size_estimates(PlannerInfo *root, RelOptInfo *rel);
+extern void set_result_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern void set_foreign_size_estimates(PlannerInfo *root, RelOptInfo *rel);
extern PathTarget *set_pathtarget_cost_width(PlannerInfo *root, PathTarget *target);
extern double compute_bitmap_pages(PlannerInfo *root, RelOptInfo *baserel,
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 81abcf5..87b6c50 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -75,8 +75,10 @@ extern MergeAppendPath *create_merge_append_path(PlannerInfo *root,
List *pathkeys,
Relids required_outer,
List *partitioned_rels);
-extern ResultPath *create_result_path(PlannerInfo *root, RelOptInfo *rel,
- PathTarget *target, List *resconstantqual);
+extern GroupResultPath *create_group_result_path(PlannerInfo *root,
+ RelOptInfo *rel,
+ PathTarget *target,
+ List *havingqual);
extern MaterialPath *create_material_path(RelOptInfo *rel, Path *subpath);
extern UniquePath *create_unique_path(PlannerInfo *root, RelOptInfo *rel,
Path *subpath, SpecialJoinInfo *sjinfo);
@@ -105,6 +107,8 @@ extern Path *create_ctescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
extern Path *create_namedtuplestorescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
+extern Path *create_resultscan_path(PlannerInfo *root, RelOptInfo *rel,
+ Relids required_outer);
extern Path *create_worktablescan_path(PlannerInfo *root, RelOptInfo *rel,
Relids required_outer);
extern ForeignPath *create_foreignscan_path(PlannerInfo *root, RelOptInfo *rel,
@@ -275,7 +279,6 @@ extern Relids min_join_parameterization(PlannerInfo *root,
Relids joinrelids,
RelOptInfo *outer_rel,
RelOptInfo *inner_rel);
-extern RelOptInfo *build_empty_join_rel(PlannerInfo *root);
extern RelOptInfo *fetch_upper_rel(PlannerInfo *root, UpperRelationKind kind,
Relids relids);
extern Relids find_childrel_parents(PlannerInfo *root, RelOptInfo *rel);
diff --git a/src/include/optimizer/prep.h b/src/include/optimizer/prep.h
index 3860877..31c6d74 100644
--- a/src/include/optimizer/prep.h
+++ b/src/include/optimizer/prep.h
@@ -21,11 +21,13 @@
/*
* prototypes for prepjointree.c
*/
+extern void replace_empty_jointree(Query *parse);
extern void pull_up_sublinks(PlannerInfo *root);
extern void inline_set_returning_functions(PlannerInfo *root);
extern void pull_up_subqueries(PlannerInfo *root);
extern void flatten_simple_union_all(PlannerInfo *root);
extern void reduce_outer_joins(PlannerInfo *root);
+extern void remove_useless_result_rtes(PlannerInfo *root);
extern Relids get_relids_in_jointree(Node *jtnode, bool include_joins);
extern Relids get_relids_for_join(PlannerInfo *root, int joinrelid);
diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out
index 1f53780..de060bb 100644
--- a/src/test/regress/expected/join.out
+++ b/src/test/regress/expected/join.out
@@ -2227,20 +2227,17 @@ explain (costs off)
select * from int8_tbl i1 left join (int8_tbl i2 join
(select 123 as x) ss on i2.q1 = x) on i1.q2 = i2.q2
order by 1, 2;
- QUERY PLAN
--------------------------------------------------
+ QUERY PLAN
+-------------------------------------------
Sort
Sort Key: i1.q1, i1.q2
-> Hash Left Join
Hash Cond: (i1.q2 = i2.q2)
-> Seq Scan on int8_tbl i1
-> Hash
- -> Hash Join
- Hash Cond: (i2.q1 = (123))
- -> Seq Scan on int8_tbl i2
- -> Hash
- -> Result
-(11 rows)
+ -> Seq Scan on int8_tbl i2
+ Filter: (q1 = 123)
+(8 rows)
select * from int8_tbl i1 left join (int8_tbl i2 join
(select 123 as x) ss on i2.q1 = x) on i1.q2 = i2.q2
@@ -3141,23 +3138,18 @@ select t1.unique2, t1.stringu1, t2.unique1, t2.stringu2 from
left join tenk1 t2
on (subq1.y1 = t2.unique1)
where t1.unique2 < 42 and t1.stringu1 > t2.stringu2;
- QUERY PLAN
------------------------------------------------------------------------
+ QUERY PLAN
+-----------------------------------------------------------------
Nested Loop
Join Filter: (t1.stringu1 > t2.stringu2)
-> Nested Loop
- Join Filter: ((0) = i1.f1)
- -> Nested Loop
- -> Nested Loop
- Join Filter: ((1) = (1))
- -> Result
- -> Result
- -> Index Scan using tenk1_unique2 on tenk1 t1
- Index Cond: ((unique2 = (11)) AND (unique2 < 42))
-> Seq Scan on int4_tbl i1
+ Filter: (f1 = 0)
+ -> Index Scan using tenk1_unique2 on tenk1 t1
+ Index Cond: ((unique2 = (11)) AND (unique2 < 42))
-> Index Scan using tenk1_unique1 on tenk1 t2
Index Cond: (unique1 = (3))
-(14 rows)
+(9 rows)
select t1.unique2, t1.stringu1, t2.unique1, t2.stringu2 from
tenk1 t1
@@ -3596,7 +3588,7 @@ select t1.* from
-> Hash Right Join
Output: i8.q2
Hash Cond: ((NULL::integer) = i8b1.q2)
- -> Hash Left Join
+ -> Hash Join
Output: i8.q2, (NULL::integer)
Hash Cond: (i8.q1 = i8b2.q1)
-> Seq Scan on public.int8_tbl i8
@@ -4018,10 +4010,10 @@ select * from
QUERY PLAN
---------------------------------------
Nested Loop Left Join
- Join Filter: ((1) = COALESCE((1)))
-> Result
-> Hash Full Join
Hash Cond: (a1.unique1 = (1))
+ Filter: (1 = COALESCE((1)))
-> Seq Scan on tenk1 a1
-> Hash
-> Result
diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out
index 588d069..a54b4a5 100644
--- a/src/test/regress/expected/subselect.out
+++ b/src/test/regress/expected/subselect.out
@@ -999,7 +999,7 @@ select * from
QUERY PLAN
----------------------------------------------------------
Subquery Scan on ss
- Output: x, u
+ Output: ss.x, ss.u
Filter: tattle(ss.x, 8)
-> ProjectSet
Output: 9, unnest('{1,2,3,11,12,13}'::integer[])
@@ -1061,7 +1061,7 @@ select * from
QUERY PLAN
----------------------------------------------------------
Subquery Scan on ss
- Output: x, u
+ Output: ss.x, ss.u
Filter: tattle(ss.x, ss.u)
-> ProjectSet
Output: 9, unnest('{1,2,3,11,12,13}'::integer[])
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: SQL/JSON revisited
@ 2023-04-04 12:36 Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 19+ messages in thread
From: Alvaro Herrera @ 2023-04-04 12:36 UTC (permalink / raw)
To: Amit Langote <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Erik Rijkers <[email protected]>; Andrew Dunstan <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; [email protected]
On 2023-Apr-04, Amit Langote wrote:
> On Tue, Apr 4, 2023 at 2:16 AM Alvaro Herrera <[email protected]> wrote:
> > - the gram.y solution to the "ON ERROR/ON EMPTY" clauses is quite ugly.
> > I think we could make that stuff use something similar to
> > ConstraintAttributeSpec with an accompanying post-processing function.
> > That would reduce the number of ad-hoc hacks, which seem excessive.
>
> Do you mean the solution involving the JsonBehavior node?
Right. It has spilled as the separate on_behavior struct in the core
parser %union in addition to the raw jsbehavior, which is something
we've gone 30 years without having, and I don't see why we should start
now.
This stuff is terrible:
json_exists_error_clause_opt:
json_exists_error_behavior ON ERROR_P { $$ = $1; }
| /* EMPTY */ { $$ = NULL; }
;
json_exists_error_behavior:
ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL); }
| TRUE_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_TRUE, NULL); }
| FALSE_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_FALSE, NULL); }
| UNKNOWN { $$ = makeJsonBehavior(JSON_BEHAVIOR_UNKNOWN, NULL); }
;
json_value_behavior:
NULL_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_NULL, NULL); }
| ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL); }
| DEFAULT a_expr { $$ = makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2); }
;
json_value_on_behavior_clause_opt:
json_value_behavior ON EMPTY_P
{ $$.on_empty = $1; $$.on_error = NULL; }
| json_value_behavior ON EMPTY_P json_value_behavior ON ERROR_P
{ $$.on_empty = $1; $$.on_error = $4; }
| json_value_behavior ON ERROR_P
{ $$.on_empty = NULL; $$.on_error = $1; }
| /* EMPTY */
{ $$.on_empty = NULL; $$.on_error = NULL; }
;
json_query_behavior:
ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL); }
| NULL_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_NULL, NULL); }
| EMPTY_P ARRAY { $$ = makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL); }
/* non-standard, for Oracle compatibility only */
| EMPTY_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL); }
| EMPTY_P OBJECT_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_EMPTY_OBJECT, NULL); }
| DEFAULT a_expr { $$ = makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2); }
;
json_query_on_behavior_clause_opt:
json_query_behavior ON EMPTY_P
{ $$.on_empty = $1; $$.on_error = NULL; }
| json_query_behavior ON EMPTY_P json_query_behavior ON ERROR_P
{ $$.on_empty = $1; $$.on_error = $4; }
| json_query_behavior ON ERROR_P
{ $$.on_empty = NULL; $$.on_error = $1; }
| /* EMPTY */
{ $$.on_empty = NULL; $$.on_error = NULL; }
;
Surely this can be made cleaner.
By the way -- that comment about clauses being non-standard, can you
spot exactly *which* clauses that comment applies to?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"El número de instalaciones de UNIX se ha elevado a 10,
y se espera que este número aumente" (UPM, 1972)
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: SQL/JSON revisited
@ 2023-04-04 19:40 Nikita Malakhov <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 19+ messages in thread
From: Nikita Malakhov @ 2023-04-04 19:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; Alexander Lakhin <[email protected]>; Erik Rijkers <[email protected]>; Andrew Dunstan <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; [email protected]
Hi hackers!
The latest SQL standard contains dot notation for JSON. Are there any plans
to include it into Pg 16?
Or maybe we should start a separate thread for it?
Thanks!
On Tue, Apr 4, 2023 at 3:36 PM Alvaro Herrera <[email protected]>
wrote:
> On 2023-Apr-04, Amit Langote wrote:
>
> > On Tue, Apr 4, 2023 at 2:16 AM Alvaro Herrera <[email protected]>
> wrote:
>
> > > - the gram.y solution to the "ON ERROR/ON EMPTY" clauses is quite ugly.
> > > I think we could make that stuff use something similar to
> > > ConstraintAttributeSpec with an accompanying post-processing
> function.
> > > That would reduce the number of ad-hoc hacks, which seem excessive.
> >
> > Do you mean the solution involving the JsonBehavior node?
>
> Right. It has spilled as the separate on_behavior struct in the core
> parser %union in addition to the raw jsbehavior, which is something
> we've gone 30 years without having, and I don't see why we should start
> now.
>
> This stuff is terrible:
>
> json_exists_error_clause_opt:
> json_exists_error_behavior ON ERROR_P { $$ = $1; }
> | /* EMPTY */ { $$ = NULL; }
> ;
>
> json_exists_error_behavior:
> ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR,
> NULL); }
> | TRUE_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_TRUE,
> NULL); }
> | FALSE_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_FALSE,
> NULL); }
> | UNKNOWN { $$ = makeJsonBehavior(JSON_BEHAVIOR_UNKNOWN,
> NULL); }
> ;
>
> json_value_behavior:
> NULL_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_NULL, NULL);
> }
> | ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR,
> NULL); }
> | DEFAULT a_expr { $$ =
> makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2); }
> ;
>
> json_value_on_behavior_clause_opt:
> json_value_behavior ON EMPTY_P
> { $$.on_empty = $1; $$.on_error =
> NULL; }
> | json_value_behavior ON EMPTY_P json_value_behavior ON ERROR_P
> { $$.on_empty = $1; $$.on_error = $4; }
> | json_value_behavior ON ERROR_P
> { $$.on_empty = NULL; $$.on_error =
> $1; }
> | /* EMPTY */
> { $$.on_empty = NULL; $$.on_error =
> NULL; }
> ;
>
> json_query_behavior:
> ERROR_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_ERROR,
> NULL); }
> | NULL_P { $$ = makeJsonBehavior(JSON_BEHAVIOR_NULL,
> NULL); }
> | EMPTY_P ARRAY { $$ =
> makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL); }
> /* non-standard, for Oracle compatibility only */
> | EMPTY_P { $$ =
> makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL); }
> | EMPTY_P OBJECT_P { $$ =
> makeJsonBehavior(JSON_BEHAVIOR_EMPTY_OBJECT, NULL); }
> | DEFAULT a_expr { $$ =
> makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2); }
> ;
>
> json_query_on_behavior_clause_opt:
> json_query_behavior ON EMPTY_P
> { $$.on_empty = $1; $$.on_error =
> NULL; }
> | json_query_behavior ON EMPTY_P json_query_behavior ON ERROR_P
> { $$.on_empty = $1; $$.on_error = $4; }
> | json_query_behavior ON ERROR_P
> { $$.on_empty = NULL; $$.on_error =
> $1; }
> | /* EMPTY */
> { $$.on_empty = NULL; $$.on_error =
> NULL; }
> ;
>
> Surely this can be made cleaner.
>
> By the way -- that comment about clauses being non-standard, can you
> spot exactly *which* clauses that comment applies to?
>
> --
> Álvaro Herrera PostgreSQL Developer —
> https://www.EnterpriseDB.com/
> "El número de instalaciones de UNIX se ha elevado a 10,
> y se espera que este número aumente" (UPM, 1972)
>
>
>
--
Regards,
Nikita Malakhov
Postgres Professional
The Russian Postgres Company
https://postgrespro.ru/
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: SQL/JSON revisited
@ 2023-04-04 19:44 Jonathan S. Katz <[email protected]>
parent: Nikita Malakhov <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Jonathan S. Katz @ 2023-04-04 19:44 UTC (permalink / raw)
To: Nikita Malakhov <[email protected]>; Alvaro Herrera <[email protected]>; +Cc: Amit Langote <[email protected]>; Alexander Lakhin <[email protected]>; Erik Rijkers <[email protected]>; Andrew Dunstan <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; [email protected]
On 4/4/23 3:40 PM, Nikita Malakhov wrote:
> Hi hackers!
>
> The latest SQL standard contains dot notation for JSON. Are there any
> plans to include it into Pg 16?
> Or maybe we should start a separate thread for it?
I would recommend starting a new thread to discuss the dot notation.
Thanks,
Jonathan
Attachments:
[application/pgp-signature] OpenPGP_signature (840B, ../../[email protected]/2-OpenPGP_signature)
download
^ permalink raw reply [nested|flat] 19+ messages in thread
* Re: SQL/JSON revisited
@ 2023-04-04 20:05 Andrew Dunstan <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 19+ messages in thread
From: Andrew Dunstan @ 2023-04-04 20:05 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; Amit Langote <[email protected]>; +Cc: Alexander Lakhin <[email protected]>; Erik Rijkers <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; vignesh C <[email protected]>; pgsql-hackers; [email protected]
On 2023-04-04 Tu 08:36, Alvaro Herrera wrote:
>
> Surely this can be made cleaner.
>
> By the way -- that comment about clauses being non-standard, can you
> spot exactly *which* clauses that comment applies to?
>
Sadly, I don't think we're going to be able to make further progress
before feature freeze. Thanks to Alvaro for advancing us a way down the
field. I hope we can get the remainder committed in the July CF.
cheers
andrew
--
Andrew Dunstan
EDB:https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v3 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 6bd8400b33b..abf6bdb2d99 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -207,7 +210,12 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
}
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -221,23 +229,24 @@ prune_freeze_xmin_is_removable(GlobalVisState *visstate, TransactionId xmin)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -245,6 +254,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -281,6 +298,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -440,7 +461,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -555,6 +576,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -612,7 +688,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -877,10 +953,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -897,7 +973,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -919,11 +996,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -967,7 +1044,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -990,7 +1067,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1017,9 +1094,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1133,11 +1210,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..45c4ae22e6a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +337,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--racicctn4wry6xe5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v4 09/19] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 155 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 134 +++++---------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 39 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 182 insertions(+), 156 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index afc5ea5e0e7..20907ba5408 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -206,23 +220,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -230,6 +245,8 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* First, initialize the new pd_prune_xid value to zero (indicating no
@@ -265,6 +282,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -400,11 +421,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -557,6 +578,72 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ /*
+ * We can use frz_conflict_horizon as our cutoff for conflicts when
+ * the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin.
+ */
+ if (!(presult->all_visible_except_removable && presult->all_frozen))
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ presult->frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(presult->frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ presult->frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -614,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -879,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -922,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -945,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -972,9 +1059,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1088,11 +1175,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index 74ebab25a95..c4553a4159c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1575,85 +1573,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.frz_conflict_horizon;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 2339abfd28a..b2a4caeb33a 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -210,9 +210,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -220,17 +221,18 @@ typedef struct PruneResult
int8 htsv[MaxHeapTuplesPerPage + 1];
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -306,6 +308,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -332,12 +335,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 042d04c8de2..b2ddc1e2549 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2179,7 +2179,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--tez7m2a73jtztiij
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0010-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v2 07/17] Execute freezing in heap_page_prune()
@ 2024-03-08 21:45 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-08 21:45 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 151 +++++++++++++++++------
src/backend/access/heap/vacuumlazy.c | 129 ++++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 41 +++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 180 insertions(+), 151 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 680a50bf8b1..5e522f5b0ba 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1046,7 +1046,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 44a5c0a917b..9c709315192 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,18 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/snapmgr.h"
#include "utils/rel.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
Relation rel;
@@ -61,17 +63,18 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
static void prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
- HeapPageFreeze *pagefrz, PruneResult *presult);
+ HeapPageFreeze *pagefrz, HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult);
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -151,15 +154,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -193,7 +196,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -207,23 +215,24 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED during
* pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* off_loc is the offset location required by the caller to use in error
* callback.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -231,6 +240,14 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ bool do_freeze;
+ int64 fpi_before = pgWalUsage.wal_fpi;
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -267,6 +284,10 @@ heap_page_prune(Relation relation, Buffer buffer,
/* for recovery conflicts */
presult->frz_conflict_horizon = InvalidTransactionId;
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
+
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(prstate.rel);
@@ -426,7 +447,7 @@ heap_page_prune(Relation relation, Buffer buffer,
if (pagefrz)
prune_prepare_freeze_tuple(page, offnum,
- pagefrz, presult);
+ pagefrz, frozen, presult);
/* Ignore items already processed as part of an earlier chain */
if (prstate.marked[offnum])
@@ -541,6 +562,61 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (presult->all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ frz_conflict_horizon = heap_frz_conflict_horizon(presult, pagefrz);
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /* Caller won't update new_relfrozenxid and new_relminmxid */
+ if (!pagefrz)
+ return;
+
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze tuples
+ * on the page, if we will set the page all-frozen in the visibility map,
+ * we can advance relfrozenxid and relminmxid to the values in
+ * pagefrz->FreezePageRelfrozenXid and pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
}
@@ -598,7 +674,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -863,10 +939,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -883,7 +959,8 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
static void
prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
HeapPageFreeze *pagefrz,
- PruneResult *presult)
+ HeapTupleFreeze *frozen,
+ PruneFreezeResult *presult)
{
bool totally_frozen;
HeapTupleHeader htup;
@@ -905,11 +982,11 @@ prune_prepare_freeze_tuple(Page page, OffsetNumber offnum,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -953,7 +1030,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -976,7 +1053,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
@@ -1003,9 +1080,9 @@ heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum)
/*
- * Perform the actual page changes needed by heap_page_prune.
- * It is expected that the caller has a full cleanup lock on the
- * buffer.
+ * Perform the actual page pruning modifications needed by
+ * heap_page_prune_and_freeze(). It is expected that the caller has a full
+ * cleanup lock on the buffer.
*/
void
heap_page_prune_execute(Buffer buffer,
@@ -1119,11 +1196,11 @@ heap_page_prune_execute(Buffer buffer,
#ifdef USE_ASSERT_CHECKING
/*
- * When heap_page_prune() was called, mark_unused_now may have been
- * passed as true, which allows would-be LP_DEAD items to be made
- * LP_UNUSED instead. This is only possible if the relation has no
- * indexes. If there are any dead items, then mark_unused_now was not
- * true and every item being marked LP_UNUSED must refer to a
+ * When heap_page_prune_and_freeze() was called, mark_unused_now may
+ * have been passed as true, which allows would-be LP_DEAD items to be
+ * made LP_UNUSED instead. This is only possible if the relation has
+ * no indexes. If there are any dead items, then mark_unused_now was
+ * not true and every item being marked LP_UNUSED must refer to a
* heap-only tuple.
*/
if (ndead > 0)
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index abbb7ab3ada..6dd8d457c9c 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -269,9 +269,6 @@ static void update_vacuum_error_info(LVRelState *vacrel,
static void restore_vacuum_error_info(LVRelState *vacrel,
const LVSavedErrInfo *saved_vacrel);
-static TransactionId heap_frz_conflict_horizon(PruneResult *presult,
- HeapPageFreeze *pagefrz);
-
/*
* heap_vacuum_rel() -- perform VACUUM for one heap relation
*
@@ -432,12 +429,13 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. lazy_scan_prune must never become confused about whether a
+ * tuple should be frozen or removed. (In the future we might want to
+ * teach lazy_scan_prune to recompute vistest from time to time, to
+ * increase the number of dead tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1379,8 +1377,8 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
* Determine the snapshotConflictHorizon for freezing. Must only be called
* after pruning and determining if the page is freezable.
*/
-static TransactionId
-heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
+TransactionId
+heap_frz_conflict_horizon(PruneFreezeResult *presult, HeapPageFreeze *pagefrz)
{
TransactionId result;
@@ -1407,21 +1405,21 @@ heap_frz_conflict_horizon(PruneResult *presult, HeapPageFreeze *pagefrz)
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1444,26 +1442,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1475,7 +1471,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1486,8 +1482,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
- &pagefrz, &presult, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and check for tuples
@@ -1604,72 +1600,23 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- snapshotConflictHorizon = heap_frz_conflict_horizon(&presult, &pagefrz);
-
/* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
+ if (presult.all_frozen)
presult.frz_conflict_horizon = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 9eea1ed315a..7bffe09fb5d 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index bea35afc4bd..69d97bb8ece 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,7 +195,7 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
@@ -204,9 +204,10 @@ typedef struct PruneResult
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -221,17 +222,18 @@ typedef struct PruneResult
/* Number of newly frozen tuples */
int nfrozen;
- /*
- * One entry for every tuple that we may freeze.
- */
- HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
- * guard against examining visibility status array members which have not yet
- * been computed.
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is
+ * meant to guard against examining visibility status array members which have
+ * not yet been computed.
*/
static inline HTSV_Result
htsv_get_valid_status(int status)
@@ -307,6 +309,9 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
+extern TransactionId heap_frz_conflict_horizon(PruneFreezeResult *presult,
+ HeapPageFreeze *pagefrz);
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -333,12 +338,12 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa7a25b8f8c..1c1a4d305d6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2175,7 +2175,7 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
-PruneResult
+PruneFreezeResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--rdqtp5puvxqotfdw
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0008-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 53 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 225 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2b7c7026429..4802d789963 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 457650ab651..e009c7579dd 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 59c81f38e51..6f9c66a872b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
-/* 'reason' codes for heap_page_prune() */
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
+
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4679660837c..cc6a33ab3ee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2192,8 +2192,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--ck6erxojvlx23byk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 52 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 224 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 6abfe36dec7..a793c0f56ee 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index eb09713311b..312695f806c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 9d047621ea5..de11c166575 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
-/* 'reason' codes for heap_page_prune() */
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfa9d5aaeac..5737bc5b945 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2191,8 +2191,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--caj67xgx3lukmr5f
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v7 08/16] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 53 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 225 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2b7c7026429..4802d789963 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1048,7 +1048,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index 457650ab651..e009c7579dd 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -594,7 +705,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -859,10 +970,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -902,7 +1013,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -925,7 +1036,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 59c81f38e51..6f9c66a872b 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
-/* 'reason' codes for heap_page_prune() */
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
+
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -332,6 +336,7 @@ extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple,
Buffer *buffer, struct TM_FailureData *tmfd);
extern void heap_inplace_update(Relation relation, HeapTuple tuple);
+
extern bool heap_prepare_freeze_tuple(HeapTupleHeader tuple,
HeapPageFreeze *pagefrz,
HeapTupleFreeze *frz, bool *totally_frozen);
@@ -358,13 +363,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 4679660837c..cc6a33ab3ee 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2192,8 +2192,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--ck6erxojvlx23byk
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
* [PATCH v9 08/21] Execute freezing in heap_page_prune()
@ 2024-03-26 00:32 Melanie Plageman <[email protected]>
0 siblings, 0 replies; 19+ messages in thread
From: Melanie Plageman @ 2024-03-26 00:32 UTC (permalink / raw)
As a step toward combining the prune and freeze WAL records, execute
freezing in heap_page_prune(). The logic to determine whether or not to
execute freeze plans was moved from lazy_scan_prune() over to
heap_page_prune() with little modification.
---
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/access/heap/pruneheap.c | 189 ++++++++++++++++++-----
src/backend/access/heap/vacuumlazy.c | 150 +++++-------------
src/backend/storage/ipc/procarray.c | 6 +-
src/include/access/heapam.h | 52 ++++---
src/tools/pgindent/typedefs.list | 2 +-
6 files changed, 224 insertions(+), 177 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 6abfe36dec7..a793c0f56ee 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1106,7 +1106,7 @@ heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin,
* We ignore unused and redirect line pointers. DEAD line pointers
* should be counted as dead, because we need vacuum to run to get rid
* of them. Note that this rule agrees with the way that
- * heap_page_prune() counts things.
+ * heap_page_prune_and_freeze() counts things.
*/
if (!ItemIdIsNormal(itemid))
{
diff --git a/src/backend/access/heap/pruneheap.c b/src/backend/access/heap/pruneheap.c
index eb09713311b..312695f806c 100644
--- a/src/backend/access/heap/pruneheap.c
+++ b/src/backend/access/heap/pruneheap.c
@@ -17,16 +17,19 @@
#include "access/heapam.h"
#include "access/heapam_xlog.h"
#include "access/htup_details.h"
+#include "access/multixact.h"
#include "access/transam.h"
#include "access/xlog.h"
+#include "commands/vacuum.h"
#include "access/xloginsert.h"
+#include "executor/instrument.h"
#include "miscadmin.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "utils/rel.h"
#include "utils/snapmgr.h"
-/* Working data for heap_page_prune and subroutines */
+/* Working data for heap_page_prune_and_freeze() and subroutines */
typedef struct
{
/* tuple visibility test, initialized for the relation */
@@ -51,6 +54,11 @@ typedef struct
* 1. Otherwise every access would need to subtract 1.
*/
bool marked[MaxHeapTuplesPerPage + 1];
+
+ /*
+ * One entry for every tuple that we may freeze.
+ */
+ HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
} PruneState;
/* Local functions */
@@ -59,14 +67,15 @@ static HTSV_Result heap_prune_satisfies_vacuum(PruneState *prstate,
Buffer buffer);
static int heap_prune_chain(Buffer buffer,
OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult);
+ PruneState *prstate, PruneFreezeResult *presult);
+
static void heap_prune_record_prunable(PruneState *prstate, TransactionId xid);
static void heap_prune_record_redirect(PruneState *prstate,
OffsetNumber offnum, OffsetNumber rdoffnum);
static void heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult);
+ PruneFreezeResult *presult);
static void heap_prune_record_unused(PruneState *prstate, OffsetNumber offnum);
static void page_verify_redirects(Page page);
@@ -146,15 +155,15 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
*/
if (PageIsFull(page) || PageGetHeapFreeSpace(page) < minfree)
{
- PruneResult presult;
+ PruneFreezeResult presult;
/*
* For now, pass mark_unused_now as false regardless of whether or
* not the relation has indexes, since we cannot safely determine
* that during on-access pruning with the current implementation.
*/
- heap_page_prune(relation, buffer, vistest, false, NULL,
- &presult, PRUNE_ON_ACCESS, NULL);
+ heap_page_prune_and_freeze(relation, buffer, vistest, false, NULL,
+ &presult, PRUNE_ON_ACCESS, NULL);
/*
* Report the number of tuples reclaimed to pgstats. This is
@@ -188,7 +197,12 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
/*
- * Prune and repair fragmentation in the specified page.
+ * Prune and repair fragmentation and potentially freeze tuples on the
+ * specified page.
+ *
+ * If the page can be marked all-frozen in the visibility map, we may
+ * opportunistically freeze tuples on the page if either its tuples are old
+ * enough or freezing will be cheap enough.
*
* Caller must have pin and buffer cleanup lock on the page. Note that we
* don't update the FSM information for page on caller's behalf. Caller might
@@ -201,12 +215,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* mark_unused_now indicates whether or not dead items can be set LP_UNUSED
* during pruning.
*
- * pagefrz contains both input and output parameters used if the caller is
- * interested in potentially freezing tuples on the page.
+ * pagefrz is an input parameter containing visibility cutoff information and
+ * the current relfrozenxid and relminmxids used if the caller is interested in
+ * freezing tuples on the page.
*
* presult contains output parameters needed by callers such as the number of
* tuples removed and the number of line pointers newly marked LP_DEAD.
- * heap_page_prune() is responsible for initializing it.
+ * heap_page_prune_and_freeze() is responsible for initializing it.
*
* reason indicates why the pruning is performed. It is included in the WAL
* record for debugging and analysis purposes, but otherwise has no effect.
@@ -215,13 +230,13 @@ heap_page_prune_opt(Relation relation, Buffer buffer)
* callback.
*/
void
-heap_page_prune(Relation relation, Buffer buffer,
- GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc)
+heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc)
{
Page page = BufferGetPage(buffer);
BlockNumber blockno = BufferGetBlockNumber(buffer);
@@ -229,6 +244,10 @@ heap_page_prune(Relation relation, Buffer buffer,
maxoff;
PruneState prstate;
HeapTupleData tup;
+ TransactionId visibility_cutoff_xid;
+ bool do_freeze;
+ bool all_visible_except_removable;
+ int64 fpi_before = pgWalUsage.wal_fpi;
/*
* Our strategy is to scan the page and make lists of items to change,
@@ -264,9 +283,20 @@ heap_page_prune(Relation relation, Buffer buffer,
* all_visible is also set to true.
*/
presult->all_frozen = true;
- presult->all_visible = true;
- /* for recovery conflicts */
- presult->visibility_cutoff_xid = InvalidTransactionId;
+
+ /*
+ * The visibility cutoff xid is the newest xmin of live tuples on the
+ * page. In the common case, this will be set as the conflict horizon the
+ * caller can use for updating the VM. If, at the end of freezing and
+ * pruning, the page is all-frozen, there is no possibility that any
+ * running transaction on the standby does not see tuples on the page as
+ * all-visible, so the conflict horizon remains InvalidTransactionId.
+ */
+ presult->vm_conflict_horizon = visibility_cutoff_xid = InvalidTransactionId;
+
+ /* For advancing relfrozenxid and relminmxid */
+ presult->new_relfrozenxid = InvalidTransactionId;
+ presult->new_relminmxid = InvalidMultiXactId;
maxoff = PageGetMaxOffsetNumber(page);
tup.t_tableOid = RelationGetRelid(relation);
@@ -291,6 +321,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* prefetching efficiency significantly / decreases the number of cache
* misses.
*/
+ all_visible_except_removable = true;
for (offnum = maxoff;
offnum >= FirstOffsetNumber;
offnum = OffsetNumberPrev(offnum))
@@ -351,13 +382,13 @@ heap_page_prune(Relation relation, Buffer buffer,
* asynchronously. See SetHintBits for more info. Check that
* the tuple is hinted xmin-committed because of that.
*/
- if (presult->all_visible)
+ if (all_visible_except_removable)
{
TransactionId xmin;
if (!HeapTupleHeaderXminCommitted(htup))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
@@ -373,25 +404,25 @@ heap_page_prune(Relation relation, Buffer buffer,
if (xmin != FrozenTransactionId &&
!GlobalVisTestIsRemovableXid(vistest, xmin))
{
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
}
/* Track newest xmin on page. */
- if (TransactionIdFollows(xmin, presult->visibility_cutoff_xid) &&
+ if (TransactionIdFollows(xmin, visibility_cutoff_xid) &&
TransactionIdIsNormal(xmin))
- presult->visibility_cutoff_xid = xmin;
+ visibility_cutoff_xid = xmin;
}
break;
case HEAPTUPLE_RECENTLY_DEAD:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_INSERT_IN_PROGRESS:
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
case HEAPTUPLE_DELETE_IN_PROGRESS:
/* This is an expected case during concurrent vacuum */
- presult->all_visible = false;
+ all_visible_except_removable = false;
break;
default:
elog(ERROR, "unexpected HeapTupleSatisfiesVacuum result");
@@ -407,11 +438,11 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Tuple with storage -- consider need to freeze */
if ((heap_prepare_freeze_tuple(htup, pagefrz,
- &presult->frozen[presult->nfrozen],
+ &prstate.frozen[presult->nfrozen],
&totally_frozen)))
{
/* Save prepared freeze plan for later */
- presult->frozen[presult->nfrozen++].offset = offnum;
+ prstate.frozen[presult->nfrozen++].offset = offnum;
}
/*
@@ -438,7 +469,7 @@ heap_page_prune(Relation relation, Buffer buffer,
* pruning and keep all_visible_except_removable to permit freezing if the
* whole page will eventually become all visible after removing tuples.
*/
- presult->all_visible_except_removable = presult->all_visible;
+ presult->all_visible = all_visible_except_removable;
/* Scan the page */
for (offnum = FirstOffsetNumber;
@@ -537,6 +568,86 @@ heap_page_prune(Relation relation, Buffer buffer,
/* Record number of newly-set-LP_DEAD items for caller */
presult->nnewlpdead = prstate.ndead;
+
+ /*
+ * Freeze the page when heap_prepare_freeze_tuple indicates that at least
+ * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
+ * freeze when pruning generated an FPI, if doing so means that we set the
+ * page all-frozen afterwards (might not happen until final heap pass).
+ */
+ if (pagefrz)
+ do_freeze = pagefrz->freeze_required ||
+ (all_visible_except_removable && presult->all_frozen &&
+ presult->nfrozen > 0 &&
+ fpi_before != pgWalUsage.wal_fpi);
+ else
+ do_freeze = false;
+
+ if (do_freeze)
+ {
+ TransactionId frz_conflict_horizon = InvalidTransactionId;
+
+ /*
+ * We can use the visibility_cutoff_xid as our cutoff for conflicts
+ * when the whole page is eligible to become all-frozen in the VM once
+ * we're done with it. Otherwise we generate a conservative cutoff by
+ * stepping back from OldestXmin. This avoids false conflicts when
+ * hot_standby_feedback is in use.
+ */
+ if (all_visible_except_removable && presult->all_frozen)
+ frz_conflict_horizon = visibility_cutoff_xid;
+ else
+ {
+ /* Avoids false conflicts when hot_standby_feedback in use */
+ frz_conflict_horizon = pagefrz->cutoffs->OldestXmin;
+ TransactionIdRetreat(frz_conflict_horizon);
+ }
+
+ /* Execute all freeze plans for page as a single atomic action */
+ heap_freeze_execute_prepared(relation, buffer,
+ frz_conflict_horizon,
+ prstate.frozen, presult->nfrozen);
+ }
+ else if (!pagefrz || !presult->all_frozen || presult->nfrozen > 0)
+ {
+ /*
+ * If we will neither freeze tuples on the page nor set the page all
+ * frozen in the visibility map, the page is not all frozen and there
+ * will be no newly frozen tuples.
+ */
+ presult->all_frozen = false;
+ presult->nfrozen = 0; /* avoid miscounts in instrumentation */
+ }
+
+ /*
+ * For callers planning to update the visibility map, the conflict horizon
+ * for that record must be the newest xmin on the page. However, if the
+ * page is completely frozen, there can be no conflict and the
+ * vm_conflict_horizon should remain InvalidTransactionId.
+ */
+ if (!presult->all_frozen)
+ presult->vm_conflict_horizon = visibility_cutoff_xid;
+
+ if (pagefrz)
+ {
+ /*
+ * If we will freeze tuples on the page or, even if we don't freeze
+ * tuples on the page, if we will set the page all-frozen in the
+ * visibility map, we can advance relfrozenxid and relminmxid to the
+ * values in pagefrz->FreezePageRelfrozenXid and
+ * pagefrz->FreezePageRelminMxid.
+ */
+ if (presult->all_frozen || presult->nfrozen > 0)
+ {
+ presult->new_relfrozenxid = pagefrz->FreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->FreezePageRelminMxid;
+ }
+ else
+ {
+ presult->new_relfrozenxid = pagefrz->NoFreezePageRelfrozenXid;
+ presult->new_relminmxid = pagefrz->NoFreezePageRelminMxid;
+ }
+ }
}
@@ -590,7 +701,7 @@ heap_prune_satisfies_vacuum(PruneState *prstate, HeapTuple tup, Buffer buffer)
*/
static int
heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
- PruneState *prstate, PruneResult *presult)
+ PruneState *prstate, PruneFreezeResult *presult)
{
int ndeleted = 0;
Page dp = (Page) BufferGetPage(buffer);
@@ -855,10 +966,10 @@ heap_prune_chain(Buffer buffer, OffsetNumber rootoffnum,
{
/*
* We found a redirect item that doesn't point to a valid follow-on
- * item. This can happen if the loop in heap_page_prune caused us to
- * visit the dead successor of a redirect item before visiting the
- * redirect item. We can clean up by setting the redirect item to
- * DEAD state or LP_UNUSED if the caller indicated.
+ * item. This can happen if the loop in heap_page_prune_and_freeze()
+ * caused us to visit the dead successor of a redirect item before
+ * visiting the redirect item. We can clean up by setting the
+ * redirect item to DEAD state or LP_UNUSED if the caller indicated.
*/
heap_prune_record_dead_or_unused(prstate, rootoffnum, presult);
}
@@ -898,7 +1009,7 @@ heap_prune_record_redirect(PruneState *prstate,
/* Record line pointer to be marked dead */
static void
heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
Assert(prstate->ndead < MaxHeapTuplesPerPage);
prstate->nowdead[prstate->ndead] = offnum;
@@ -921,7 +1032,7 @@ heap_prune_record_dead(PruneState *prstate, OffsetNumber offnum,
*/
static void
heap_prune_record_dead_or_unused(PruneState *prstate, OffsetNumber offnum,
- PruneResult *presult)
+ PruneFreezeResult *presult)
{
/*
* If the caller set mark_unused_now to true, we can remove dead tuples
diff --git a/src/backend/access/heap/vacuumlazy.c b/src/backend/access/heap/vacuumlazy.c
index f474e661428..8beef4093ae 100644
--- a/src/backend/access/heap/vacuumlazy.c
+++ b/src/backend/access/heap/vacuumlazy.c
@@ -430,12 +430,12 @@ heap_vacuum_rel(Relation rel, VacuumParams *params,
* as an upper bound on the XIDs stored in the pages we'll actually scan
* (NewRelfrozenXid tracking must never be allowed to miss unfrozen XIDs).
*
- * Next acquire vistest, a related cutoff that's used in heap_page_prune.
- * We expect vistest will always make heap_page_prune remove any deleted
- * tuple whose xmax is < OldestXmin. lazy_scan_prune must never become
- * confused about whether a tuple should be frozen or removed. (In the
- * future we might want to teach lazy_scan_prune to recompute vistest from
- * time to time, to increase the number of dead tuples it can prune away.)
+ * Next acquire vistest, a related cutoff that's used in
+ * heap_page_prune_and_freeze(). We expect vistest will always make
+ * heap_page_prune_and_freeze() remove any deleted tuple whose xmax is <
+ * OldestXmin. (In the future we might want to teach lazy_scan_prune to
+ * recompute vistest from time to time, to increase the number of dead
+ * tuples it can prune away.)
*/
vacrel->aggressive = vacuum_get_cutoffs(rel, params, &vacrel->cutoffs);
vacrel->rel_pages = orig_rel_pages = RelationGetNumberOfBlocks(rel);
@@ -1378,21 +1378,21 @@ lazy_scan_new_or_empty(LVRelState *vacrel, Buffer buf, BlockNumber blkno,
*
* Caller must hold pin and buffer cleanup lock on the buffer.
*
- * Prior to PostgreSQL 14 there were very rare cases where heap_page_prune()
- * was allowed to disagree with our HeapTupleSatisfiesVacuum() call about
- * whether or not a tuple should be considered DEAD. This happened when an
- * inserting transaction concurrently aborted (after our heap_page_prune()
- * call, before our HeapTupleSatisfiesVacuum() call). There was rather a lot
- * of complexity just so we could deal with tuples that were DEAD to VACUUM,
- * but nevertheless were left with storage after pruning.
+ * Prior to PostgreSQL 14 there were very rare cases where
+ * heap_page_prune_and_freeze() was allowed to disagree with our
+ * HeapTupleSatisfiesVacuum() call about whether or not a tuple should be
+ * considered DEAD. This happened when an inserting transaction concurrently
+ * aborted (after our heap_page_prune_and_freeze() call, before our
+ * HeapTupleSatisfiesVacuum() call). There was rather a lot of complexity just
+ * so we could deal with tuples that were DEAD to VACUUM, but nevertheless were
+ * left with storage after pruning.
*
* As of Postgres 17, we circumvent this problem altogether by reusing the
- * result of heap_page_prune()'s visibility check. Without the second call to
- * HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and there can be no
- * disagreement. We'll just handle such tuples as if they had become fully dead
- * right after this operation completes instead of in the middle of it. Note that
- * any tuple that becomes dead after the call to heap_page_prune() can't need to
- * be frozen, because it was visible to another session when vacuum started.
+ * result of heap_page_prune_and_freeze()'s visibility check. Without the
+ * second call to HeapTupleSatisfiesVacuum(), there is no new HTSV_Result and
+ * there can be no disagreement. We'll just handle such tuples as if they had
+ * become fully dead right after this operation completes instead of in the
+ * middle of it.
*
* vmbuffer is the buffer containing the VM block with visibility information
* for the heap block, blkno. all_visible_according_to_vm is the saved
@@ -1415,26 +1415,24 @@ lazy_scan_prune(LVRelState *vacrel,
OffsetNumber offnum,
maxoff;
ItemId itemid;
- PruneResult presult;
+ PruneFreezeResult presult;
int lpdead_items,
live_tuples,
recently_dead_tuples;
HeapPageFreeze pagefrz;
bool hastup = false;
- bool do_freeze;
- int64 fpi_before = pgWalUsage.wal_fpi;
OffsetNumber deadoffsets[MaxHeapTuplesPerPage];
Assert(BufferGetBlockNumber(buf) == blkno);
/*
* maxoff might be reduced following line pointer array truncation in
- * heap_page_prune. That's safe for us to ignore, since the reclaimed
- * space will continue to look like LP_UNUSED items below.
+ * heap_page_prune_and_freeze(). That's safe for us to ignore, since the
+ * reclaimed space will continue to look like LP_UNUSED items below.
*/
maxoff = PageGetMaxOffsetNumber(page);
- /* Initialize (or reset) page-level state */
+ /* Initialize pagefrz */
pagefrz.freeze_required = false;
pagefrz.FreezePageRelfrozenXid = vacrel->NewRelfrozenXid;
pagefrz.FreezePageRelminMxid = vacrel->NewRelminMxid;
@@ -1446,7 +1444,7 @@ lazy_scan_prune(LVRelState *vacrel,
recently_dead_tuples = 0;
/*
- * Prune all HOT-update chains in this page.
+ * Prune all HOT-update chains and potentially freeze tuples on this page.
*
* We count the number of tuples removed from the page by the pruning step
* in presult.ndeleted. It should not be confused with lpdead_items;
@@ -1457,8 +1455,8 @@ lazy_scan_prune(LVRelState *vacrel,
* items LP_UNUSED, so mark_unused_now should be true if no indexes and
* false otherwise.
*/
- heap_page_prune(rel, buf, vacrel->vistest, vacrel->nindexes == 0, &pagefrz,
- &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
+ heap_page_prune_and_freeze(rel, buf, vacrel->vistest, vacrel->nindexes == 0,
+ &pagefrz, &presult, PRUNE_VACUUM_SCAN, &vacrel->offnum);
/*
* Now scan the page to collect LP_DEAD items and update the variables set
@@ -1571,86 +1569,20 @@ lazy_scan_prune(LVRelState *vacrel,
vacrel->offnum = InvalidOffsetNumber;
- /*
- * Freeze the page when heap_prepare_freeze_tuple indicates that at least
- * one XID/MXID from before FreezeLimit/MultiXactCutoff is present. Also
- * freeze when pruning generated an FPI, if doing so means that we set the
- * page all-frozen afterwards (might not happen until final heap pass).
- */
- do_freeze = pagefrz.freeze_required ||
- (presult.all_visible_except_removable && presult.all_frozen &&
- presult.nfrozen > 0 &&
- fpi_before != pgWalUsage.wal_fpi);
+ Assert(MultiXactIdIsValid(presult.new_relminmxid));
+ vacrel->NewRelfrozenXid = presult.new_relfrozenxid;
+ Assert(TransactionIdIsValid(presult.new_relfrozenxid));
+ vacrel->NewRelminMxid = presult.new_relminmxid;
- if (do_freeze)
+ if (presult.nfrozen > 0)
{
- TransactionId snapshotConflictHorizon;
-
/*
- * We're freezing the page. Our final NewRelfrozenXid doesn't need to
- * be affected by the XIDs that are just about to be frozen anyway.
+ * We never increment the frozen_pages instrumentation counter when
+ * nfrozen == 0, since it only counts pages with newly frozen tuples
+ * (don't confuse that with pages newly set all-frozen in VM).
*/
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
-
vacrel->frozen_pages++;
- /*
- * We can use frz_conflict_horizon as our cutoff for conflicts when
- * the whole page is eligible to become all-frozen in the VM once
- * we're done with it. Otherwise we generate a conservative cutoff by
- * stepping back from OldestXmin.
- */
- if (presult.all_visible_except_removable && presult.all_frozen)
- snapshotConflictHorizon = presult.visibility_cutoff_xid;
- else
- {
- /* Avoids false conflicts when hot_standby_feedback in use */
- snapshotConflictHorizon = pagefrz.cutoffs->OldestXmin;
- TransactionIdRetreat(snapshotConflictHorizon);
- }
-
- /* Using same cutoff when setting VM is now unnecessary */
- if (presult.all_visible_except_removable && presult.all_frozen)
- presult.visibility_cutoff_xid = InvalidTransactionId;
-
- /* Execute all freeze plans for page as a single atomic action */
- heap_freeze_execute_prepared(vacrel->rel, buf,
- snapshotConflictHorizon,
- presult.frozen, presult.nfrozen);
-
- }
- else if (presult.all_frozen && presult.nfrozen == 0)
- {
- /* Page should be all visible except to-be-removed tuples */
- Assert(presult.all_visible_except_removable);
-
- /*
- * We have no freeze plans to execute, so there's no added cost from
- * following the freeze path. That's why it was chosen. This is
- * important in the case where the page only contains totally frozen
- * tuples at this point (perhaps only following pruning). Such pages
- * can be marked all-frozen in the VM by our caller, even though none
- * of its tuples were newly frozen here (note that the "no freeze"
- * path never sets pages all-frozen).
- *
- * We never increment the frozen_pages instrumentation counter here,
- * since it only counts pages with newly frozen tuples (don't confuse
- * that with pages newly set all-frozen in VM).
- */
- vacrel->NewRelfrozenXid = pagefrz.FreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.FreezePageRelminMxid;
- }
- else
- {
- /*
- * Page requires "no freeze" processing. It might be set all-visible
- * in the visibility map, but it can never be set all-frozen.
- */
- vacrel->NewRelfrozenXid = pagefrz.NoFreezePageRelfrozenXid;
- vacrel->NewRelminMxid = pagefrz.NoFreezePageRelminMxid;
- presult.all_frozen = false;
- presult.nfrozen = 0; /* avoid miscounts in instrumentation */
}
/*
@@ -1676,7 +1608,7 @@ lazy_scan_prune(LVRelState *vacrel,
Assert(presult.all_frozen == debug_all_frozen);
Assert(!TransactionIdIsValid(debug_cutoff) ||
- debug_cutoff == presult.visibility_cutoff_xid);
+ debug_cutoff == presult.vm_conflict_horizon);
}
#endif
@@ -1730,7 +1662,7 @@ lazy_scan_prune(LVRelState *vacrel,
if (presult.all_frozen)
{
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
flags |= VISIBILITYMAP_ALL_FROZEN;
}
@@ -1750,7 +1682,7 @@ lazy_scan_prune(LVRelState *vacrel,
PageSetAllVisible(page);
MarkBufferDirty(buf);
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
- vmbuffer, presult.visibility_cutoff_xid,
+ vmbuffer, presult.vm_conflict_horizon,
flags);
}
@@ -1815,11 +1747,11 @@ lazy_scan_prune(LVRelState *vacrel,
/*
* Set the page all-frozen (and all-visible) in the VM.
*
- * We can pass InvalidTransactionId as our visibility_cutoff_xid,
- * since a snapshotConflictHorizon sufficient to make everything safe
- * for REDO was logged when the page's tuples were frozen.
+ * We can pass InvalidTransactionId as our vm_conflict_horizon, since
+ * a snapshotConflictHorizon sufficient to make everything safe for
+ * REDO was logged when the page's tuples were frozen.
*/
- Assert(!TransactionIdIsValid(presult.visibility_cutoff_xid));
+ Assert(!TransactionIdIsValid(presult.vm_conflict_horizon));
visibilitymap_set(vacrel->rel, blkno, buf, InvalidXLogRecPtr,
vmbuffer, InvalidTransactionId,
VISIBILITYMAP_ALL_VISIBLE |
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index b3cd248fb64..88a6d504dff 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1715,9 +1715,9 @@ TransactionIdIsActive(TransactionId xid)
* Note: the approximate horizons (see definition of GlobalVisState) are
* updated by the computations done here. That's currently required for
* correctness and a small optimization. Without doing so it's possible that
- * heap vacuum's call to heap_page_prune() uses a more conservative horizon
- * than later when deciding which tuples can be removed - which the code
- * doesn't expect (breaking HOT).
+ * heap vacuum's call to heap_page_prune_and_freeze() uses a more conservative
+ * horizon than later when deciding which tuples can be removed - which the
+ * code doesn't expect (breaking HOT).
*/
static void
ComputeXidHorizons(ComputeXidHorizonsResult *h)
diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h
index 9d047621ea5..de11c166575 100644
--- a/src/include/access/heapam.h
+++ b/src/include/access/heapam.h
@@ -195,13 +195,13 @@ typedef struct HeapPageFreeze
/*
* Per-page state returned from pruning
*/
-typedef struct PruneResult
+typedef struct PruneFreezeResult
{
int ndeleted; /* Number of tuples deleted from the page */
int nnewlpdead; /* Number of newly LP_DEAD items */
/*
- * The rest of the fields in PruneResult are only guaranteed to be
+ * The rest of the fields in PruneFreezeResult are only guaranteed to be
* initialized if heap_page_prune is passed PruneReason VACUUM_SCAN.
*/
@@ -212,23 +212,22 @@ typedef struct PruneResult
*/
bool all_visible;
- /*
- * Whether or not the page is all-visible except for tuples which will be
- * removed during vacuum's second pass. This is used by VACUUM to
- * determine whether or not to consider opportunistically freezing the
- * page.
- */
- bool all_visible_except_removable;
-
/* Whether or not the page can be set all-frozen in the VM */
bool all_frozen;
- TransactionId visibility_cutoff_xid; /* Newest xmin on the page */
+
+ /*
+ * If the page is all-visible and not all-frozen this is the oldest xid
+ * that can see the page as all-visible. It is to be used as the snapshot
+ * conflict horizon when emitting a XLOG_HEAP2_VISIBLE record.
+ */
+ TransactionId vm_conflict_horizon;
/*
* Tuple visibility is only computed once for each tuple, for correctness
- * and efficiency reasons; see comment in heap_page_prune() for details.
- * This is of type int8[], instead of HTSV_Result[], so we can use -1 to
- * indicate no visibility has been computed, e.g. for LP_DEAD items.
+ * and efficiency reasons; see comment in heap_page_prune_and_freeze() for
+ * details. This is of type int8[], instead of HTSV_Result[], so we can
+ * use -1 to indicate no visibility has been computed, e.g. for LP_DEAD
+ * items.
*
* This needs to be MaxHeapTuplesPerPage + 1 long as FirstOffsetNumber is
* 1. Otherwise every access would need to subtract 1.
@@ -242,9 +241,14 @@ typedef struct PruneResult
* One entry for every tuple that we may freeze.
*/
HeapTupleFreeze frozen[MaxHeapTuplesPerPage];
-} PruneResult;
+ /* New value of relfrozenxid found by heap_page_prune_and_freeze() */
+ TransactionId new_relfrozenxid;
+
+ /* New value of relminmxid found by heap_page_prune_and_freeze() */
+ MultiXactId new_relminmxid;
+} PruneFreezeResult;
-/* 'reason' codes for heap_page_prune() */
+/* 'reason' codes for heap_page_prune_and_freeze() */
typedef enum
{
PRUNE_ON_ACCESS, /* on-access pruning */
@@ -254,7 +258,7 @@ typedef enum
/*
* Pruning calculates tuple visibility once and saves the results in an array
- * of int8. See PruneResult.htsv for details. This helper function is meant to
+ * of int8. See PruneFreezeResult.htsv for details. This helper function is meant to
* guard against examining visibility status array members which have not yet
* been computed.
*/
@@ -361,13 +365,13 @@ extern TransactionId heap_index_delete_tuples(Relation rel,
/* in heap/pruneheap.c */
struct GlobalVisState;
extern void heap_page_prune_opt(Relation relation, Buffer buffer);
-extern void heap_page_prune(Relation relation, Buffer buffer,
- struct GlobalVisState *vistest,
- bool mark_unused_now,
- HeapPageFreeze *pagefrz,
- PruneResult *presult,
- PruneReason reason,
- OffsetNumber *off_loc);
+extern void heap_page_prune_and_freeze(Relation relation, Buffer buffer,
+ struct GlobalVisState *vistest,
+ bool mark_unused_now,
+ HeapPageFreeze *pagefrz,
+ PruneFreezeResult *presult,
+ PruneReason reason,
+ OffsetNumber *off_loc);
extern void heap_page_prune_execute(Buffer buffer, bool lp_truncate_only,
OffsetNumber *redirected, int nredirected,
OffsetNumber *nowdead, int ndead,
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cfa9d5aaeac..5737bc5b945 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2191,8 +2191,8 @@ ProjectionPath
PromptInterruptContext
ProtocolVersion
PrsStorage
+PruneFreezeResult
PruneReason
-PruneResult
PruneState
PruneStepResult
PsqlScanCallbacks
--
2.40.1
--caj67xgx3lukmr5f
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v9-0009-Make-opp-freeze-heuristic-compatible-with-prune-f.patch"
^ permalink raw reply [nested|flat] 19+ messages in thread
end of thread, other threads:[~2024-03-26 00:32 UTC | newest]
Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-14 22:04 Re: "SELECT ... FROM DUAL" is not quite as silly as it appears Tom Lane <[email protected]>
2018-10-15 03:25 ` Tom Lane <[email protected]>
2023-04-04 12:36 Re: SQL/JSON revisited Alvaro Herrera <[email protected]>
2023-04-04 19:40 ` Re: SQL/JSON revisited Nikita Malakhov <[email protected]>
2023-04-04 19:44 ` Re: SQL/JSON revisited Jonathan S. Katz <[email protected]>
2023-04-04 20:05 ` Re: SQL/JSON revisited Andrew Dunstan <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v3 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v4 09/19] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-08 21:45 [PATCH v2 07/17] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v9 08/21] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
2024-03-26 00:32 [PATCH v7 08/16] Execute freezing in heap_page_prune() Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox