agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v5 2/2] ALTER TABLE ... DETACH CONCURRENTLY 10+ messages / 2 participants [nested] [flat]
* [PATCH v5 2/2] ALTER TABLE ... DETACH CONCURRENTLY @ 2020-08-03 23:21 Alvaro Herrera <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Alvaro Herrera @ 2020-08-03 23:21 UTC (permalink / raw) --- doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/alter_table.sgml | 25 +- src/backend/catalog/heap.c | 13 +- src/backend/catalog/index.c | 4 +- src/backend/catalog/partition.c | 28 +- src/backend/catalog/pg_inherits.c | 51 +- src/backend/commands/indexcmds.c | 7 +- src/backend/commands/tablecmds.c | 568 ++++++++++++++---- src/backend/commands/trigger.c | 5 +- src/backend/executor/execPartition.c | 29 +- src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/optimizer/util/plancat.c | 11 +- src/backend/parser/gram.y | 23 +- src/backend/partitioning/partbounds.c | 6 +- src/backend/partitioning/partdesc.c | 21 +- src/backend/tcop/utility.c | 19 + src/backend/utils/cache/partcache.c | 15 +- src/bin/psql/describe.c | 40 +- src/include/catalog/pg_inherits.h | 7 +- src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/partitioning/partdesc.h | 5 +- src/include/utils/snapmgr.h | 1 + .../detach-partition-concurrently-1.out | 301 ++++++++++ .../detach-partition-concurrently-2.out | 66 ++ src/test/isolation/isolation_schedule | 2 + .../detach-partition-concurrently-1.spec | 90 +++ .../detach-partition-concurrently-2.spec | 41 ++ src/test/modules/delay_execution/Makefile | 3 +- .../expected/partition-removal-1.out | 79 +++ .../specs/partition-removal-1.spec | 45 ++ 32 files changed, 1347 insertions(+), 173 deletions(-) create mode 100644 src/test/isolation/expected/detach-partition-concurrently-1.out create mode 100644 src/test/isolation/expected/detach-partition-concurrently-2.out create mode 100644 src/test/isolation/specs/detach-partition-concurrently-1.spec create mode 100644 src/test/isolation/specs/detach-partition-concurrently-2.spec create mode 100644 src/test/modules/delay_execution/expected/partition-removal-1.out create mode 100644 src/test/modules/delay_execution/specs/partition-removal-1.spec diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 3a2266526c..759dd8da5b 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -4482,6 +4482,16 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l when using declarative partitioning. </para></entry> </row> + + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>inhdetached</structfield> <type>bool</type> + </para> + <para> + Set to true for a partition that is in the process of being detached; + false otherwise. + </para></entry> + </row> </tbody> </tgroup> </table> diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index c25ef5abd6..09673dcf1c 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -36,7 +36,9 @@ ALTER TABLE ALL IN TABLESPACE <replaceable class="parameter">name</replaceable> ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> ATTACH PARTITION <replaceable class="parameter">partition_name</replaceable> { FOR VALUES <replaceable class="parameter">partition_bound_spec</replaceable> | DEFAULT } ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> - DETACH PARTITION <replaceable class="parameter">partition_name</replaceable> + DETACH PARTITION <replaceable class="parameter">partition_name</replaceable> [ CONCURRENTLY ] +ALTER TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> + DETACH PARTITION <replaceable class="parameter">partition_name</replaceable> FINALIZE <phrase>where <replaceable class="parameter">action</replaceable> is one of:</phrase> @@ -938,7 +940,8 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM </varlistentry> <varlistentry> - <term><literal>DETACH PARTITION</literal> <replaceable class="parameter">partition_name</replaceable></term> + <term><literal>DETACH PARTITION</literal> <replaceable class="parameter">partition_name</replaceable> [ { CONCURRENTLY | FINALIZE } ]</term> + <listitem> <para> This form detaches the specified partition of the target table. The detached @@ -947,6 +950,24 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM attached to the target table's indexes are detached. Any triggers that were created as clones of those in the target table are removed. </para> + <para> + If <literal>CONCURRENTLY</literal> is specified, this process runs in two + transactions in order to avoid blocking other sessions that might be accessing + the partitioned table. During the first transaction, a + <literal>SHARE UPDATE EXCLUSIVE</literal> lock is taken on both parent table and + partition, and its partition is marked detached; at that point, the transaction + is committed and all transactions using the partitioned table are waited for. + Once all those transactions are gone, the second stage acquires + <literal>ACCESS EXCLUSIVE</literal> on the partition, and the detach process + completes. + <literal>CONCURRENTLY</literal> is not allowed if the + partitioned table contains a default partition. + </para> + <para> + If <literal>FINALIZE</literal> is specified, a previous + <literal>DETACH CONCURRENTLY</literal> invocation that was cancelled or + interrupted is completed. + </para> </listitem> </varlistentry> diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 21f2240ade..73e26fa400 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -2546,10 +2546,12 @@ StoreConstraints(Relation rel, List *cooked_constraints, bool is_internal) * Returns a list of CookedConstraint nodes that shows the cooked form of * the default and constraint expressions added to the relation. * - * NB: caller should have opened rel with AccessExclusiveLock, and should - * hold that lock till end of transaction. Also, we assume the caller has - * done a CommandCounterIncrement if necessary to make the relation's catalog - * tuples visible. + * NB: caller should have opened rel with some self-conflicting lock mode, + * and should hold that lock till end of transaction; for normal cases that'll + * be AccessExclusiveLock, but if caller knows that the constraint is already + * enforced by some other means, it can be ShareUpdateExclusiveLock. Also, we + * assume the caller has done a CommandCounterIncrement if necessary to make + * the relation's catalog tuples visible. */ List * AddRelationNewConstraints(Relation rel, @@ -3818,7 +3820,8 @@ StorePartitionBound(Relation rel, Relation parent, PartitionBoundSpec *bound) * relcache entry for that partition every time a partition is added or * removed. */ - defaultPartOid = get_default_oid_from_partdesc(RelationGetPartitionDesc(parent)); + defaultPartOid = + get_default_oid_from_partdesc(RelationGetPartitionDesc(parent, false)); if (OidIsValid(defaultPartOid)) CacheInvalidateRelcacheByRelid(defaultPartOid); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index cffbc0ac38..0dc53b25b8 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1831,7 +1831,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName) List *ancestors = get_partition_ancestors(oldIndexId); Oid parentIndexRelid = linitial_oid(ancestors); - DeleteInheritsTuple(oldIndexId, parentIndexRelid); + DeleteInheritsTuple(oldIndexId, parentIndexRelid, false, NULL); StoreSingleInheritance(newIndexId, parentIndexRelid, 1); list_free(ancestors); @@ -2425,7 +2425,7 @@ index_drop(Oid indexId, bool concurrent, bool concurrent_lock_mode) /* * fix INHERITS relation */ - DeleteInheritsTuple(indexId, InvalidOid); + DeleteInheritsTuple(indexId, InvalidOid, false, NULL); /* * We are presently too lazy to attempt to compute the new correct value diff --git a/src/backend/catalog/partition.c b/src/backend/catalog/partition.c index af7754d6ab..4327a0b150 100644 --- a/src/backend/catalog/partition.c +++ b/src/backend/catalog/partition.c @@ -32,7 +32,7 @@ #include "utils/rel.h" #include "utils/syscache.h" -static Oid get_partition_parent_worker(Relation inhRel, Oid relid); +static Oid get_partition_parent_worker(Relation inhRel, Oid relid, bool *detached); static void get_partition_ancestors_worker(Relation inhRel, Oid relid, List **ancestors); @@ -42,6 +42,9 @@ static void get_partition_ancestors_worker(Relation inhRel, Oid relid, * * Returns inheritance parent of a partition by scanning pg_inherits * + * If the partition is in the process of being detached, InvalidOid is + * returned and no error is thrown. + * * Note: Because this function assumes that the relation whose OID is passed * as an argument will have precisely one parent, it should only be called * when it is known that the relation is a partition. @@ -51,12 +54,15 @@ get_partition_parent(Oid relid) { Relation catalogRelation; Oid result; + bool detached; catalogRelation = table_open(InheritsRelationId, AccessShareLock); - result = get_partition_parent_worker(catalogRelation, relid); + result = get_partition_parent_worker(catalogRelation, relid, &detached); - if (!OidIsValid(result)) + if (detached) + result = InvalidOid; + else if (!OidIsValid(result)) elog(ERROR, "could not find tuple for parent of relation %u", relid); table_close(catalogRelation, AccessShareLock); @@ -70,13 +76,15 @@ get_partition_parent(Oid relid) * given relation */ static Oid -get_partition_parent_worker(Relation inhRel, Oid relid) +get_partition_parent_worker(Relation inhRel, Oid relid, bool *detached) { SysScanDesc scan; ScanKeyData key[2]; Oid result = InvalidOid; HeapTuple tuple; + *detached = false; + ScanKeyInit(&key[0], Anum_pg_inherits_inhrelid, BTEqualStrategyNumber, F_OIDEQ, @@ -93,7 +101,14 @@ get_partition_parent_worker(Relation inhRel, Oid relid) { Form_pg_inherits form = (Form_pg_inherits) GETSTRUCT(tuple); - result = form->inhparent; + /* A partition being detached has no parent */ + if (form->inhdetached) + { + *detached = true; + result = InvalidOid; + } + else + result = form->inhparent; } systable_endscan(scan); @@ -134,9 +149,10 @@ static void get_partition_ancestors_worker(Relation inhRel, Oid relid, List **ancestors) { Oid parentOid; + bool detached; /* Recursion ends at the topmost level, ie., when there's no parent */ - parentOid = get_partition_parent_worker(inhRel, relid); + parentOid = get_partition_parent_worker(inhRel, relid, &detached); if (parentOid == InvalidOid) return; diff --git a/src/backend/catalog/pg_inherits.c b/src/backend/catalog/pg_inherits.c index f3783961b7..3aec885288 100644 --- a/src/backend/catalog/pg_inherits.c +++ b/src/backend/catalog/pg_inherits.c @@ -52,7 +52,8 @@ typedef struct SeenRelsEntry * against possible DROPs of child relations. */ List * -find_inheritance_children(Oid parentrelId, LOCKMODE lockmode) +find_inheritance_children(Oid parentrelId, bool include_detached, + LOCKMODE lockmode) { List *list = NIL; Relation relation; @@ -91,7 +92,13 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode) while ((inheritsTuple = systable_getnext(scan)) != NULL) { + /* skip detached at caller's request */ + if (((Form_pg_inherits) GETSTRUCT(inheritsTuple))->inhdetached && + !include_detached) + continue; + inhrelid = ((Form_pg_inherits) GETSTRUCT(inheritsTuple))->inhrelid; + if (numoids >= maxoids) { maxoids *= 2; @@ -160,6 +167,9 @@ find_inheritance_children(Oid parentrelId, LOCKMODE lockmode) * given rel; caller should already have locked it). If lockmode is NoLock * then no locks are acquired, but caller must beware of race conditions * against possible DROPs of child relations. + * + * NB - No current callers of this routine are interested in children being + * concurrently detached, so there's no provision to include them. */ List * find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents) @@ -199,7 +209,8 @@ find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **numparents) ListCell *lc; /* Get the direct children of this rel */ - currentchildren = find_inheritance_children(currentrel, lockmode); + currentchildren = find_inheritance_children(currentrel, false, + lockmode); /* * Add to the queue only those children not already seen. This avoids @@ -428,6 +439,7 @@ StoreSingleInheritance(Oid relationId, Oid parentOid, int32 seqNumber) values[Anum_pg_inherits_inhrelid - 1] = ObjectIdGetDatum(relationId); values[Anum_pg_inherits_inhparent - 1] = ObjectIdGetDatum(parentOid); values[Anum_pg_inherits_inhseqno - 1] = Int32GetDatum(seqNumber); + values[Anum_pg_inherits_inhdetached - 1] = BoolGetDatum(false); memset(nulls, 0, sizeof(nulls)); @@ -447,10 +459,21 @@ StoreSingleInheritance(Oid relationId, Oid parentOid, int32 seqNumber) * as InvalidOid, in which case all tuples matching inhrelid are deleted; * otherwise only delete tuples with the specified inhparent. * + * 'detached' is the expected state of the inhdetached flag. If the catalog + * row does not match that state, an error is raised. When used on + * partitions, the partition name must be passed, for possible error messages. + * + * If allow_detached is passed false, then an error is raised if the child is + * already marked detached. A name must be supplied in that case, for the + * error message. This should only be used with table partitions. + * XXX the name bit is pretty weird and problematic for non-partition cases; + * what if the flag check fails? + * * Returns whether at least one row was deleted. */ bool -DeleteInheritsTuple(Oid inhrelid, Oid inhparent) +DeleteInheritsTuple(Oid inhrelid, Oid inhparent, bool detached, + const char *childname) { bool found = false; Relation catalogRelation; @@ -477,6 +500,28 @@ DeleteInheritsTuple(Oid inhrelid, Oid inhparent) parent = ((Form_pg_inherits) GETSTRUCT(inheritsTuple))->inhparent; if (!OidIsValid(inhparent) || parent == inhparent) { + bool inhdetached; + + inhdetached = ((Form_pg_inherits) GETSTRUCT(inheritsTuple))->inhdetached; + + /* + * Raise error depending on state. This should only happen for + * partitions, but we have no way to cross-check. + */ + if (inhdetached && !detached) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot detach partition \"%s\"", + childname), + errdetail("The partition is being detached concurrently or has an unfinished detach."), + errhint("Use ALTER TABLE ... DETACH PARTITION ... FINALIZE to complete the pending detach operation"))); + if (!inhdetached && detached) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot complete detaching partition \"%s\"", + childname), + errdetail("There's no pending concurrent detach."))); + CatalogTupleDelete(catalogRelation, &inheritsTuple->t_self); found = true; } diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 992f4813b4..ee4af9ea32 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -401,7 +401,7 @@ CompareOpclassOptions(Datum *opts1, Datum *opts2, int natts) * GetCurrentVirtualXIDs. If, during any iteration, a particular vxid * doesn't show up in the output, we know we can forget about it. */ -static void +void WaitForOlderSnapshots(TransactionId limitXmin, bool progress) { int n_old_snapshots; @@ -1114,7 +1114,7 @@ DefineIndex(Oid relationId, */ if (partitioned && stmt->relation && !stmt->relation->inh) { - PartitionDesc pd = RelationGetPartitionDesc(rel); + PartitionDesc pd = RelationGetPartitionDesc(rel, false); if (pd->nparts != 0) flags |= INDEX_CREATE_INVALID; @@ -1171,7 +1171,7 @@ DefineIndex(Oid relationId, * * If we're called internally (no stmt->relation), recurse always. */ - partdesc = RelationGetPartitionDesc(rel); + partdesc = RelationGetPartitionDesc(rel, false); if ((!stmt->relation || stmt->relation->inh) && partdesc->nparts > 0) { int nparts = partdesc->nparts; @@ -3849,6 +3849,7 @@ IndexSetParentIndex(Relation partitionIdx, Oid parentOid) values[Anum_pg_inherits_inhparent - 1] = ObjectIdGetDatum(parentOid); values[Anum_pg_inherits_inhseqno - 1] = Int32GetDatum(1); + values[Anum_pg_inherits_inhdetached - 1] = BoolGetDatum(false); memset(isnull, false, sizeof(isnull)); tuple = heap_form_tuple(RelationGetDescr(pg_inherits), diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index a8528a3423..d7b9c63e5f 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -542,7 +542,8 @@ static PartitionSpec *transformPartitionSpec(Relation rel, PartitionSpec *partsp static void ComputePartitionAttrs(ParseState *pstate, Relation rel, List *partParams, AttrNumber *partattrs, List **partexprs, Oid *partopclass, Oid *partcollation, char strategy); static void CreateInheritance(Relation child_rel, Relation parent_rel); -static void RemoveInheritance(Relation child_rel, Relation parent_rel); +static void RemoveInheritance(Relation child_rel, Relation parent_rel, + bool allow_detached); static ObjectAddress ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd, AlterTableUtilityContext *context); @@ -551,8 +552,14 @@ static void QueuePartitionConstraintValidation(List **wqueue, Relation scanrel, List *partConstraint, bool validate_default); static void CloneRowTriggersToPartition(Relation parent, Relation partition); +static void DetachAddConstraintIfNeeded(List **wqueue, Relation partRel); static void DropClonedTriggersFromPartition(Oid partitionId); -static ObjectAddress ATExecDetachPartition(Relation rel, RangeVar *name); +static ObjectAddress ATExecDetachPartition(List **wqueue, AlteredTableInfo *tab, + Relation rel, RangeVar *name, + bool concurrent); +static void DetachPartitionFinalize(Relation rel, Relation partRel, + bool concurrent, Oid defaultPartOid, bool wait); +static ObjectAddress ATExecDetachPartitionFinalize(Relation rel, RangeVar *name); static ObjectAddress ATExecAttachPartitionIdx(List **wqueue, Relation rel, RangeVar *name); static void validatePartitionedIndex(Relation partedIdx, Relation partedTbl); @@ -987,7 +994,8 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, * lock the partition so as to avoid a deadlock. */ defaultPartOid = - get_default_oid_from_partdesc(RelationGetPartitionDesc(parent)); + get_default_oid_from_partdesc(RelationGetPartitionDesc(parent, + false)); if (OidIsValid(defaultPartOid)) defaultRel = table_open(defaultPartOid, AccessExclusiveLock); @@ -3158,7 +3166,7 @@ renameatt_internal(Oid myrelid, * expected_parents will only be 0 if we are not already recursing. */ if (expected_parents == 0 && - find_inheritance_children(myrelid, NoLock) != NIL) + find_inheritance_children(myrelid, false, NoLock) != NIL) ereport(ERROR, (errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("inherited column \"%s\" must be renamed in child tables too", @@ -3357,7 +3365,7 @@ rename_constraint_internal(Oid myrelid, else { if (expected_parents == 0 && - find_inheritance_children(myrelid, NoLock) != NIL) + find_inheritance_children(myrelid, false, NoLock) != NIL) ereport(ERROR, (errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("inherited constraint \"%s\" must be renamed in child tables too", @@ -3976,7 +3984,14 @@ AlterTableGetLockLevel(List *cmds) break; case AT_DetachPartition: - cmd_lockmode = AccessExclusiveLock; + if (((PartitionCmd *) cmd->def)->concurrent) + cmd_lockmode = ShareUpdateExclusiveLock; + else + cmd_lockmode = AccessExclusiveLock; + break; + + case AT_DetachPartitionFinalize: + cmd_lockmode = ShareUpdateExclusiveLock; break; case AT_CheckNotNull: @@ -4367,6 +4382,11 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, /* No command-specific prep needed */ pass = AT_PASS_MISC; break; + case AT_DetachPartitionFinalize: + ATSimplePermissions(rel, ATT_TABLE); + /* No command-specific prep needed */ + pass = AT_PASS_MISC; + break; default: /* oops */ elog(ERROR, "unrecognized alter table type: %d", (int) cmd->subtype); @@ -4752,7 +4772,12 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Assert(cmd != NULL); /* ATPrepCmd ensures it must be a table */ Assert(rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE); - ATExecDetachPartition(rel, ((PartitionCmd *) cmd->def)->name); + ATExecDetachPartition(wqueue, tab, rel, + ((PartitionCmd *) cmd->def)->name, + ((PartitionCmd *) cmd->def)->concurrent); + break; + case AT_DetachPartitionFinalize: + ATExecDetachPartitionFinalize(rel, ((PartitionCmd *) cmd->def)->name); break; case AT_AlterCollationRefreshVersion: /* ATPrepCmd ensured it must be an index */ @@ -6184,7 +6209,7 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, */ if (colDef->identity && recurse && - find_inheritance_children(myrelid, NoLock) != NIL) + find_inheritance_children(myrelid, false, NoLock) != NIL) ereport(ERROR, (errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("cannot recursively add identity column to table that has child tables"))); @@ -6417,7 +6442,8 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, * routines, we have to do this one level of recursion at a time; we can't * use find_all_inheritors to do it in one pass. */ - children = find_inheritance_children(RelationGetRelid(rel), lockmode); + children = + find_inheritance_children(RelationGetRelid(rel), false, lockmode); /* * If we are told not to recurse, there had better not be any child @@ -6571,7 +6597,7 @@ ATPrepDropNotNull(Relation rel, bool recurse, bool recursing) */ if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { - PartitionDesc partdesc = RelationGetPartitionDesc(rel); + PartitionDesc partdesc = RelationGetPartitionDesc(rel, false); Assert(partdesc != NULL); if (partdesc->nparts > 0 && !recurse && !recursing) @@ -6671,17 +6697,22 @@ ATExecDropNotNull(Relation rel, const char *colName, LOCKMODE lockmode) if (rel->rd_rel->relispartition) { Oid parentId = get_partition_parent(RelationGetRelid(rel)); - Relation parent = table_open(parentId, AccessShareLock); - TupleDesc tupDesc = RelationGetDescr(parent); + Relation parent; + TupleDesc tupDesc; AttrNumber parent_attnum; - parent_attnum = get_attnum(parentId, colName); - if (TupleDescAttr(tupDesc, parent_attnum - 1)->attnotnull) - ereport(ERROR, - (errcode(ERRCODE_INVALID_TABLE_DEFINITION), - errmsg("column \"%s\" is marked NOT NULL in parent table", - colName))); - table_close(parent, AccessShareLock); + if (parentId != InvalidOid) + { + parent = table_open(parentId, AccessShareLock); + tupDesc = RelationGetDescr(parent); + parent_attnum = get_attnum(parentId, colName); + if (TupleDescAttr(tupDesc, parent_attnum - 1)->attnotnull) + ereport(ERROR, + (errcode(ERRCODE_INVALID_TABLE_DEFINITION), + errmsg("column \"%s\" is marked NOT NULL in parent table", + colName))); + table_close(parent, AccessShareLock); + } } /* @@ -7280,7 +7311,7 @@ ATPrepDropExpression(Relation rel, AlterTableCmd *cmd, bool recurse, bool recurs * resulting state can be properly dumped and restored. */ if (!recurse && - find_inheritance_children(RelationGetRelid(rel), lockmode)) + find_inheritance_children(RelationGetRelid(rel), false, lockmode)) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("ALTER TABLE / DROP EXPRESSION must be applied to child tables too"))); @@ -7863,7 +7894,8 @@ ATExecDropColumn(List **wqueue, Relation rel, const char *colName, * routines, we have to do this one level of recursion at a time; we can't * use find_all_inheritors to do it in one pass. */ - children = find_inheritance_children(RelationGetRelid(rel), lockmode); + children = + find_inheritance_children(RelationGetRelid(rel), false, lockmode); if (children) { @@ -8327,7 +8359,8 @@ ATAddCheckConstraint(List **wqueue, AlteredTableInfo *tab, Relation rel, * routines, we have to do this one level of recursion at a time; we can't * use find_all_inheritors to do it in one pass. */ - children = find_inheritance_children(RelationGetRelid(rel), lockmode); + children = + find_inheritance_children(RelationGetRelid(rel), false, lockmode); /* * Check if ONLY was specified with ALTER TABLE. If so, allow the @@ -8942,7 +8975,7 @@ addFkRecurseReferenced(List **wqueue, Constraint *fkconstraint, Relation rel, */ if (pkrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { - PartitionDesc pd = RelationGetPartitionDesc(pkrel); + PartitionDesc pd = RelationGetPartitionDesc(pkrel, false); for (int i = 0; i < pd->nparts; i++) { @@ -9076,7 +9109,7 @@ addFkRecurseReferencing(List **wqueue, Constraint *fkconstraint, Relation rel, } else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { - PartitionDesc pd = RelationGetPartitionDesc(rel); + PartitionDesc pd = RelationGetPartitionDesc(rel, false); /* * Recurse to take appropriate action on each partition; either we @@ -10860,7 +10893,8 @@ ATExecDropConstraint(Relation rel, const char *constrName, * use find_all_inheritors to do it in one pass. */ if (!is_no_inherit_constraint) - children = find_inheritance_children(RelationGetRelid(rel), lockmode); + children = + find_inheritance_children(RelationGetRelid(rel), false, lockmode); else children = NIL; @@ -11244,7 +11278,8 @@ ATPrepAlterColumnType(List **wqueue, } } else if (!recursing && - find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL) + find_inheritance_children(RelationGetRelid(rel), false, + NoLock) != NIL) ereport(ERROR, (errcode(ERRCODE_INVALID_TABLE_DEFINITION), errmsg("type of inherited column \"%s\" must be changed in child tables too", @@ -14139,7 +14174,7 @@ ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode) */ /* Off to RemoveInheritance() where most of the work happens */ - RemoveInheritance(rel, parent_rel); + RemoveInheritance(rel, parent_rel, false); ObjectAddressSet(address, RelationRelationId, RelationGetRelid(parent_rel)); @@ -14150,12 +14185,70 @@ ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode) return address; } +/* + * MarkInheritDetached + * + * When a partition is detached from its parent concurrently, we don't + * remove the pg_inherits row until a second transaction; as a preparatory + * step, this function marks the entry as 'detached', so that other + * transactions know to ignore the partition. + */ +static void +MarkInheritDetached(Relation child_rel, Relation parent_rel) +{ + Relation catalogRelation; + SysScanDesc scan; + ScanKeyData key; + HeapTuple inheritsTuple; + bool found = false; + + /* + * Find pg_inherits entries by inhrelid. + */ + catalogRelation = table_open(InheritsRelationId, RowExclusiveLock); + ScanKeyInit(&key, + Anum_pg_inherits_inhrelid, + BTEqualStrategyNumber, F_OIDEQ, + ObjectIdGetDatum(RelationGetRelid(child_rel))); + scan = systable_beginscan(catalogRelation, InheritsRelidSeqnoIndexId, + true, NULL, 1, &key); + + while (HeapTupleIsValid(inheritsTuple = systable_getnext(scan))) + { + HeapTuple newtup; + + if (((Form_pg_inherits) GETSTRUCT(inheritsTuple))->inhparent != + RelationGetRelid(parent_rel)) + elog(ERROR, "bad parent tuple found for partition %u", + RelationGetRelid(child_rel)); + + newtup = heap_copytuple(inheritsTuple); + ((Form_pg_inherits) GETSTRUCT(newtup))->inhdetached = true; + + CatalogTupleUpdate(catalogRelation, + &inheritsTuple->t_self, + newtup); + found = true; + } + + /* Done */ + systable_endscan(scan); + table_close(catalogRelation, RowExclusiveLock); + + if (!found) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_TABLE), + errmsg("relation \"%s\" is not a partition of relation \"%s\"", + RelationGetRelationName(child_rel), + RelationGetRelationName(parent_rel)))); +} + /* * RemoveInheritance * * Drop a parent from the child's parents. This just adjusts the attinhcount * and attislocal of the columns and removes the pg_inherit and pg_depend - * entries. + * entries. expect_detached is passed down to DeleteInheritsTuple, q.v.. * * If attinhcount goes to 0 then attislocal gets set to true. If it goes back * up attislocal stays true, which means if a child is ever removed from a @@ -14169,7 +14262,7 @@ ATExecDropInherit(Relation rel, RangeVar *parent, LOCKMODE lockmode) * Common to ATExecDropInherit() and ATExecDetachPartition(). */ static void -RemoveInheritance(Relation child_rel, Relation parent_rel) +RemoveInheritance(Relation child_rel, Relation parent_rel, bool expect_detached) { Relation catalogRelation; SysScanDesc scan; @@ -14185,7 +14278,9 @@ RemoveInheritance(Relation child_rel, Relation parent_rel) child_is_partition = true; found = DeleteInheritsTuple(RelationGetRelid(child_rel), - RelationGetRelid(parent_rel)); + RelationGetRelid(parent_rel), + expect_detached, + RelationGetRelationName(child_rel)); if (!found) { if (child_is_partition) @@ -16293,7 +16388,7 @@ QueuePartitionConstraintValidation(List **wqueue, Relation scanrel, } else if (scanrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) { - PartitionDesc partdesc = RelationGetPartitionDesc(scanrel); + PartitionDesc partdesc = RelationGetPartitionDesc(scanrel, false); int i; for (i = 0; i < partdesc->nparts; i++) @@ -16353,7 +16448,7 @@ ATExecAttachPartition(List **wqueue, Relation rel, PartitionCmd *cmd, * new partition will change its partition constraint. */ defaultPartOid = - get_default_oid_from_partdesc(RelationGetPartitionDesc(rel)); + get_default_oid_from_partdesc(RelationGetPartitionDesc(rel, false)); if (OidIsValid(defaultPartOid)) LockRelationOid(defaultPartOid, AccessExclusiveLock); @@ -16942,105 +17037,230 @@ CloneRowTriggersToPartition(Relation parent, Relation partition) * ALTER TABLE DETACH PARTITION * * Return the address of the relation that is no longer a partition of rel. + * + * If concurrent mode is requested, we run in two transactions. A side- + * effect is that this command cannot run in a multi-part ALTER TABLE. + * Currently, that's enforced by the grammar. + * + * The strategy for concurrency is to first modify the partition catalog + * rows to make it visible to everyone that the partition is detached, + * lock the partition against writes, and commit the transaction; anyone + * who requests the partition descriptor from that point onwards has to + * ignore such a partition. In a second transaction, we wait until all + * transactions that could have seen the partition as attached are gone, + * then we remove the rest of partition metadata (pg_inherits and + * pg_class.relpartbounds). */ static ObjectAddress -ATExecDetachPartition(Relation rel, RangeVar *name) +ATExecDetachPartition(List **wqueue, AlteredTableInfo *tab, Relation rel, + RangeVar *name, bool concurrent) { - Relation partRel, - classRel; - HeapTuple tuple, - newtuple; - Datum new_val[Natts_pg_class]; - bool new_null[Natts_pg_class], - new_repl[Natts_pg_class]; + Relation partRel; ObjectAddress address; Oid defaultPartOid; - List *indexes; - List *fks; - ListCell *cell; /* * We must lock the default partition, because detaching this partition * will change its partition constraint. */ defaultPartOid = - get_default_oid_from_partdesc(RelationGetPartitionDesc(rel)); - if (OidIsValid(defaultPartOid)) - LockRelationOid(defaultPartOid, AccessExclusiveLock); - - partRel = table_openrv(name, ShareUpdateExclusiveLock); - - /* Ensure that foreign keys still hold after this detach */ - ATDetachCheckNoForeignKeyRefs(partRel); - - /* All inheritance related checks are performed within the function */ - RemoveInheritance(partRel, rel); - - /* Update pg_class tuple */ - classRel = table_open(RelationRelationId, RowExclusiveLock); - tuple = SearchSysCacheCopy1(RELOID, - ObjectIdGetDatum(RelationGetRelid(partRel))); - if (!HeapTupleIsValid(tuple)) - elog(ERROR, "cache lookup failed for relation %u", - RelationGetRelid(partRel)); - Assert(((Form_pg_class) GETSTRUCT(tuple))->relispartition); - - /* Clear relpartbound and reset relispartition */ - memset(new_val, 0, sizeof(new_val)); - memset(new_null, false, sizeof(new_null)); - memset(new_repl, false, sizeof(new_repl)); - new_val[Anum_pg_class_relpartbound - 1] = (Datum) 0; - new_null[Anum_pg_class_relpartbound - 1] = true; - new_repl[Anum_pg_class_relpartbound - 1] = true; - newtuple = heap_modify_tuple(tuple, RelationGetDescr(classRel), - new_val, new_null, new_repl); - - ((Form_pg_class) GETSTRUCT(newtuple))->relispartition = false; - CatalogTupleUpdate(classRel, &newtuple->t_self, newtuple); - heap_freetuple(newtuple); - + get_default_oid_from_partdesc(RelationGetPartitionDesc(rel, false)); if (OidIsValid(defaultPartOid)) { /* - * If the relation being detached is the default partition itself, - * remove it from the parent's pg_partitioned_table entry. + * Concurrent detaching when a default partition exists is not + * supported. The main problem is that the default partition + * constraint would change. And there's a definitional problem: what + * should happen to the tuples that are being inserted that belong to + * the partition being detached? Putting them on the partition being + * detached would be wrong, since they'd become "lost" after the but + * we cannot put them in the default partition either until we alter + * its partition constraint. * - * If not, we must invalidate default partition's relcache entry, as - * in StorePartitionBound: its partition constraint depends on every - * other partition's partition constraint. + * I think we could solve this problem if we effected the constraint + * change before committing the first transaction. But the lock would + * have to remain AEL and it would cause concurrent query planning to + * be blocked, so changing it that way would be even worse. */ - if (RelationGetRelid(partRel) == defaultPartOid) - update_default_partition_oid(RelationGetRelid(rel), InvalidOid); - else - CacheInvalidateRelcacheByRelid(defaultPartOid); + if (concurrent) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot detach partitions concurrently when a default partition exists"))); + LockRelationOid(defaultPartOid, AccessExclusiveLock); } - /* detach indexes too */ - indexes = RelationGetIndexList(partRel); - foreach(cell, indexes) + /* + * In concurrent mode, the partition is locked with share-update-exclusive + * in the first transaction. This allows concurrent transactions to be + * doing DML to the partition. + */ + partRel = table_openrv(name, concurrent ? ShareLock : + AccessExclusiveLock); + + /* + * Check inheritance conditions and either delete the pg_inherits row + * (in non-concurrent mode) or just set the inhisdetached flag. + */ + if (!concurrent) + RemoveInheritance(partRel, rel, false); + else + MarkInheritDetached(partRel, rel); + + /* + * Ensure that foreign keys still hold after this detach. This keeps + * locks on the referencing tables, which prevents concurrent transactions + * from adding rows that we wouldn't see. For this to work in concurrent + * mode, it is critical that the partition appears as no longer attached + * for the RI queries as soon as the first transaction commits. + */ + ATDetachCheckNoForeignKeyRefs(partRel); + + /* + * Concurrent mode has to work harder; first we add a new constraint to the + * partition that matches the partition constraint, if there isn't a matching + * one already. The reason for this is that the planner may have made + * optimizations that depend on the constraint. XXX Isn't it sufficient to + * invalidate the partition's relcache entry? + * + * Then we close our existing transaction, and in a new one wait for + * all processes to catch up on the catalog updates we've done so far; at + * that point we can complete the operation. + */ + if (concurrent) { - Oid idxid = lfirst_oid(cell); - Relation idx; - Oid constrOid; + Oid partrelid, + parentrelid; + LOCKTAG tag; + char *parentrelname; + char *partrelname; - if (!has_superclass(idxid)) - continue; + /* Add constraint, if needed. XXX hopefully we can just remove this */ + DetachAddConstraintIfNeeded(wqueue, partRel); - Assert((IndexGetRelation(get_partition_parent(idxid), false) == - RelationGetRelid(rel))); + /* + * We're almost done now; the only traces that remain are the + * pg_inherits tuple and the partition's relpartbounds. Before we can + * remove those, we need to wait until all transactions that know that + * this is a partition are gone. + */ - idx = index_open(idxid, AccessExclusiveLock); - IndexSetParentIndex(idx, InvalidOid); + /* + * Remember relation OIDs to re-acquire them later; and relation names + * too, for error messages if something is dropped in between. + */ + partrelid = RelationGetRelid(partRel); + parentrelid = RelationGetRelid(rel); + parentrelname = MemoryContextStrdup(PortalContext, + RelationGetRelationName(rel)); + partrelname = MemoryContextStrdup(PortalContext, + RelationGetRelationName(partRel)); - /* If there's a constraint associated with the index, detach it too */ - constrOid = get_relation_idx_constraint_oid(RelationGetRelid(partRel), - idxid); - if (OidIsValid(constrOid)) - ConstraintSetParentConstraint(constrOid, InvalidOid, InvalidOid); + /* Invalidate relcache entries for the parent -- must be before close */ + CacheInvalidateRelcache(rel); + + table_close(partRel, NoLock); + table_close(rel, NoLock); + tab->rel = NULL; + + /* Make updated catalog entry visible */ + PopActiveSnapshot(); + CommitTransactionCommand(); + + StartTransactionCommand(); + + /* + * Now wait. This ensures that all queries that were planned including + * the partition are finished before we remove the rest of catalog + * entries. We don't need or indeed want to acquire this lock, though + * -- that would block later queries. + * + * We don't need to concern ourselves with waiting for a lock on the + * partition itself, since we will acquire AccessExclusiveLock below. + */ + SET_LOCKTAG_RELATION(tag, MyDatabaseId, parentrelid); + WaitForLockersMultiple(list_make1(&tag), AccessExclusiveLock, false); + + /* + * Now acquire locks in both relations again. Note they may have been + * removed in the meantime, so care is required. + */ + rel = try_relation_open(parentrelid, ShareUpdateExclusiveLock); + partRel = try_relation_open(partrelid, AccessExclusiveLock); + + /* If the relations aren't there, something bad happened; bail out */ + if (rel == NULL) + { + if (partRel != NULL) /* shouldn't happen */ + elog(WARNING, "dangling partition \"%s\" remains, can't fix", + partrelname); + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("partitioned table \"%s\" was removed concurrently", + parentrelname))); + } + if (partRel == NULL) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("partition \"%s\" was removed concurrently", partrelname))); + + tab->rel = rel; - index_close(idx, NoLock); } - table_close(classRel, RowExclusiveLock); + + /* Do the final part of detaching */ + DetachPartitionFinalize(rel, partRel, concurrent, defaultPartOid, false); + + ObjectAddressSet(address, RelationRelationId, RelationGetRelid(partRel)); + + /* keep our lock until commit */ + table_close(partRel, NoLock); + + return address; +} + +/* + * Second part of ALTER TABLE .. DETACH. + * + * This is separate so that it can be run independently when the second + * transaction of the concurrent algorithm fails (crash or abort). + */ +static void +DetachPartitionFinalize(Relation rel, Relation partRel, bool concurrent, + Oid defaultPartOid, bool wait) +{ + Relation classRel; + List *fks; + ListCell *cell; + List *indexes; + Datum new_val[Natts_pg_class]; + bool new_null[Natts_pg_class], + new_repl[Natts_pg_class]; + HeapTuple tuple, + newtuple; + + /* + * If asked to, wait until existing snapshots are gone. This is important + * if the second transaction of DETACH PARTITION CONCURRENTLY is canceled: + * the user could immediately run DETACH FINALIZE without actually waiting + * for existing transactions. We must not complete the detach action until + * all such queries are complete (otherwise we would present them with an + * inconsistent view of catalogs). + */ + if (wait) + { + Snapshot snap = GetActiveSnapshot(); + + WaitForOlderSnapshots(snap->xmin, false); + } + + if (concurrent) + { + /* + * We can remove the pg_inherits row now. (In the non-concurrent case, + * this was already done). + */ + RemoveInheritance(partRel, rel, true); + } /* Drop any triggers that were cloned on creation/attach. */ DropClonedTriggersFromPartition(RelationGetRelid(partRel)); @@ -17113,22 +17333,150 @@ ATExecDetachPartition(Relation rel, RangeVar *name) ObjectAddressSet(constraint, ConstraintRelationId, constrOid); performDeletion(&constraint, DROP_RESTRICT, 0); } - CommandCounterIncrement(); + + /* Now we can detach indexes */ + indexes = RelationGetIndexList(partRel); + foreach(cell, indexes) + { + Oid idxid = lfirst_oid(cell); + Relation idx; + Oid constrOid; + + if (!has_superclass(idxid)) + continue; + + Assert((IndexGetRelation(get_partition_parent(idxid), false) == + RelationGetRelid(rel))); + + idx = index_open(idxid, AccessExclusiveLock); + IndexSetParentIndex(idx, InvalidOid); + + /* If there's a constraint associated with the index, detach it too */ + constrOid = get_relation_idx_constraint_oid(RelationGetRelid(partRel), + idxid); + if (OidIsValid(constrOid)) + ConstraintSetParentConstraint(constrOid, InvalidOid, InvalidOid); + + index_close(idx, NoLock); + } + + /* Update pg_class tuple */ + classRel = table_open(RelationRelationId, RowExclusiveLock); + tuple = SearchSysCacheCopy1(RELOID, + ObjectIdGetDatum(RelationGetRelid(partRel))); + if (!HeapTupleIsValid(tuple)) + elog(ERROR, "cache lookup failed for relation %u", + RelationGetRelid(partRel)); + Assert(((Form_pg_class) GETSTRUCT(tuple))->relispartition); + + /* Clear relpartbound and reset relispartition */ + memset(new_val, 0, sizeof(new_val)); + memset(new_null, false, sizeof(new_null)); + memset(new_repl, false, sizeof(new_repl)); + new_val[Anum_pg_class_relpartbound - 1] = (Datum) 0; + new_null[Anum_pg_class_relpartbound - 1] = true; + new_repl[Anum_pg_class_relpartbound - 1] = true; + newtuple = heap_modify_tuple(tuple, RelationGetDescr(classRel), + new_val, new_null, new_repl); + + ((Form_pg_class) GETSTRUCT(newtuple))->relispartition = false; + CatalogTupleUpdate(classRel, &newtuple->t_self, newtuple); + heap_freetuple(newtuple); + table_close(classRel, RowExclusiveLock); + + if (OidIsValid(defaultPartOid)) + { + /* + * If the relation being detached is the default partition itself, + * remove it from the parent's pg_partitioned_table entry. + * + * If not, we must invalidate default partition's relcache entry, as + * in StorePartitionBound: its partition constraint depends on every + * other partition's partition constraint. + */ + if (RelationGetRelid(partRel) == defaultPartOid) + update_default_partition_oid(RelationGetRelid(rel), InvalidOid); + else + CacheInvalidateRelcacheByRelid(defaultPartOid); + } /* * Invalidate the parent's relcache so that the partition is no longer * included in its partition descriptor. */ CacheInvalidateRelcache(rel); +} + +/* + * ALTER TABLE ... DETACH PARTITION ... FINALIZE + * + * To use when a DETACH PARTITION command previously did not run to + * completion; this completes the detaching process. + */ +static ObjectAddress +ATExecDetachPartitionFinalize(Relation rel, RangeVar *name) +{ + Relation partRel; + ObjectAddress address; + + partRel = table_openrv(name, AccessExclusiveLock); + + DetachPartitionFinalize(rel, partRel, true, InvalidOid, true); ObjectAddressSet(address, RelationRelationId, RelationGetRelid(partRel)); - /* keep our lock until commit */ table_close(partRel, NoLock); return address; } +/* + * DetachAddConstraintIfNeeded + * Subroutine for ATExecDetachPartition. Create a constraint that + * takes the place of the partition constraint, but avoid creating + * a dupe if an equivalent constraint already exists. + */ +static void +DetachAddConstraintIfNeeded(List **wqueue, Relation partRel) +{ + AlteredTableInfo *tab; + Expr *constraintExpr; + TupleDesc td = RelationGetDescr(partRel); + Constraint *n; + + constraintExpr = make_ands_explicit(RelationGetPartitionQual(partRel)); + + /* If an identical constraint exists, we don't need to create one */ + if (td->constr && td->constr->num_check > 0) + { + for (int i = 0; i < td->constr->num_check; i++) + { + Node *thisconstr; + + thisconstr = stringToNode(td->constr->check[i].ccbin); + + if (equal(constraintExpr, thisconstr)) + return; + } + } + + tab = ATGetQueueEntry(wqueue, partRel); + + /* Add constraint on partition, equivalent to the partition constraint */ + n = makeNode(Constraint); + n->contype = CONSTR_CHECK; + n->conname = NULL; + n->location = -1; + n->is_no_inherit = false; + n->raw_expr = NULL; + n->cooked_expr = nodeToString(constraintExpr); + n->initially_valid = true; + n->skip_validation = true; + /* It's a re-add, since it nominally already exists */ + ATAddCheckConstraint(wqueue, tab, partRel, n, + true, false, true, ShareUpdateExclusiveLock); +} + /* * DropClonedTriggersFromPartition * subroutine for ATExecDetachPartition to remove any triggers that were @@ -17324,7 +17672,7 @@ ATExecAttachPartitionIdx(List **wqueue, Relation parentIdx, RangeVar *name) RelationGetRelationName(partIdx)))); /* Make sure it indexes a partition of the other index's table */ - partDesc = RelationGetPartitionDesc(parentTbl); + partDesc = RelationGetPartitionDesc(parentTbl, false); found = false; for (i = 0; i < partDesc->nparts; i++) { @@ -17478,7 +17826,7 @@ validatePartitionedIndex(Relation partedIdx, Relation partedTbl) * If we found as many inherited indexes as the partitioned table has * partitions, we're good; update pg_index to set indisvalid. */ - if (tuples == RelationGetPartitionDesc(partedTbl)->nparts) + if (tuples == RelationGetPartitionDesc(partedTbl, false)->nparts) { Relation idxRel; HeapTuple newtup; diff --git a/src/backend/commands/trigger.c b/src/backend/commands/trigger.c index 12229364f1..d422d332d5 100644 --- a/src/backend/commands/trigger.c +++ b/src/backend/commands/trigger.c @@ -1129,7 +1129,7 @@ CreateTrigger(CreateTrigStmt *stmt, const char *queryString, */ if (partition_recurse) { - PartitionDesc partdesc = RelationGetPartitionDesc(rel); + PartitionDesc partdesc = RelationGetPartitionDesc(rel, false); List *idxs = NIL; List *childTbls = NIL; ListCell *l; @@ -1151,7 +1151,8 @@ CreateTrigger(CreateTrigStmt *stmt, const char *queryString, ListCell *l; List *idxs = NIL; - idxs = find_inheritance_children(indexOid, ShareRowExclusiveLock); + idxs = find_inheritance_children(indexOid, false, + ShareRowExclusiveLock); foreach(l, idxs) childTbls = lappend_oid(childTbls, IndexGetRelation(lfirst_oid(l), diff --git a/src/backend/executor/execPartition.c b/src/backend/executor/execPartition.c index 941731a0a9..87336d81e0 100644 --- a/src/backend/executor/execPartition.c +++ b/src/backend/executor/execPartition.c @@ -567,6 +567,7 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, EState *estate, int partidx) { ModifyTable *node = (ModifyTable *) mtstate->ps.plan; + Oid partOid = dispatch->partdesc->oids[partidx]; Relation rootrel = rootResultRelInfo->ri_RelationDesc, partrel; Relation firstResultRel = mtstate->resultRelInfo[0].ri_RelationDesc; @@ -577,7 +578,7 @@ ExecInitPartitionInfo(ModifyTableState *mtstate, EState *estate, oldcxt = MemoryContextSwitchTo(proute->memcxt); - partrel = table_open(dispatch->partdesc->oids[partidx], RowExclusiveLock); + partrel = table_open(partOid, RowExclusiveLock); leaf_part_rri = makeNode(ResultRelInfo); InitResultRelInfo(leaf_part_rri, @@ -1046,9 +1047,21 @@ ExecInitPartitionDispatchInfo(EState *estate, int dispatchidx; MemoryContext oldcxt; + /* + * For data modification, it is better that executor does not include + * partitions being detached, except in snapshot-isolation mode. This + * means that a read-committed transaction immediately gets a "no + * partition for tuple" error when a tuple is inserted into a partition + * that's being detached concurrently, but a transaction in repeatable- + * read mode can still use the partition. Note that because partition + * detach uses ShareLock on the partition (which conflicts with DML), + * we're certain that the detach won't be able to complete until any + * inserting transaction is done. + */ if (estate->es_partition_directory == NULL) estate->es_partition_directory = - CreatePartitionDirectory(estate->es_query_cxt); + CreatePartitionDirectory(estate->es_query_cxt, + IsolationUsesXactSnapshot()); oldcxt = MemoryContextSwitchTo(proute->memcxt); @@ -1628,9 +1641,10 @@ ExecCreatePartitionPruneState(PlanState *planstate, ListCell *lc; int i; + /* Executor must always include detached partitions */ if (estate->es_partition_directory == NULL) estate->es_partition_directory = - CreatePartitionDirectory(estate->es_query_cxt); + CreatePartitionDirectory(estate->es_query_cxt, true); n_part_hierarchies = list_length(partitionpruneinfo->prune_infos); Assert(n_part_hierarchies > 0); @@ -1696,9 +1710,12 @@ ExecCreatePartitionPruneState(PlanState *planstate, partrel); /* - * Initialize the subplan_map and subpart_map. Since detaching a - * partition requires AccessExclusiveLock, no partitions can have - * disappeared, nor can the bounds for any partition have changed. + * Initialize the subplan_map and subpart_map. + * + * Because we request detached partitions to be included, and + * detaching waits for old transactions, it is safe to assume that + * no partitions have disappeared since this query was planned. + * * However, new partitions may have been added. */ Assert(partdesc->nparts >= pinfo->nparts); diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index ba3ccc712c..80685bdff9 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -4678,6 +4678,7 @@ _copyPartitionCmd(const PartitionCmd *from) COPY_NODE_FIELD(name); COPY_NODE_FIELD(bound); + COPY_SCALAR_FIELD(concurrent); return newnode; } diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index a2ef853dc2..1e73308280 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2941,6 +2941,7 @@ _equalPartitionCmd(const PartitionCmd *a, const PartitionCmd *b) { COMPARE_NODE_FIELD(name); COMPARE_NODE_FIELD(bound); + COMPARE_SCALAR_FIELD(concurrent); return true; } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index da322b453e..da5a8c5f5a 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -2135,10 +2135,17 @@ set_relation_partition_info(PlannerInfo *root, RelOptInfo *rel, { PartitionDesc partdesc; - /* Create the PartitionDirectory infrastructure if we didn't already */ + /* + * Create the PartitionDirectory infrastructure if we didn't already. + * In snapshot-isolation transactions, we always include detached + * partitions, because otherwise they become invisible too soon. + */ if (root->glob->partition_directory == NULL) + { root->glob->partition_directory = - CreatePartitionDirectory(CurrentMemoryContext); + CreatePartitionDirectory(CurrentMemoryContext, + IsolationUsesXactSnapshot()); + } partdesc = PartitionDirectoryLookup(root->glob->partition_directory, relation); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 31c95443a5..4ae8caf79f 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -644,7 +644,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); EXCLUDE EXCLUDING EXCLUSIVE EXECUTE EXISTS EXPLAIN EXPRESSION EXTENSION EXTERNAL EXTRACT - FALSE_P FAMILY FETCH FILTER FIRST_P FLOAT_P FOLLOWING FOR + FALSE_P FAMILY FETCH FILTER FINALIZE FIRST_P FLOAT_P FOLLOWING FOR FORCE FOREIGN FORWARD FREEZE FROM FULL FUNCTION FUNCTIONS GENERATED GLOBAL GRANT GRANTED GREATEST GROUP_P GROUPING GROUPS @@ -2094,12 +2094,13 @@ partition_cmd: n->subtype = AT_AttachPartition; cmd->name = $3; cmd->bound = $4; + cmd->concurrent = false; n->def = (Node *) cmd; $$ = (Node *) n; } - /* ALTER TABLE <name> DETACH PARTITION <partition_name> */ - | DETACH PARTITION qualified_name + /* ALTER TABLE <name> DETACH PARTITION <partition_name> [CONCURRENTLY] */ + | DETACH PARTITION qualified_name opt_concurrently { AlterTableCmd *n = makeNode(AlterTableCmd); PartitionCmd *cmd = makeNode(PartitionCmd); @@ -2107,8 +2108,21 @@ partition_cmd: n->subtype = AT_DetachPartition; cmd->name = $3; cmd->bound = NULL; + cmd->concurrent = $4; n->def = (Node *) cmd; + $$ = (Node *) n; + } + | DETACH PARTITION qualified_name FINALIZE + { + AlterTableCmd *n = makeNode(AlterTableCmd); + PartitionCmd *cmd = makeNode(PartitionCmd); + + n->subtype = AT_DetachPartitionFinalize; + cmd->name = $3; + cmd->bound = NULL; + cmd->concurrent = false; + n->def = (Node *) cmd; $$ = (Node *) n; } ; @@ -2123,6 +2137,7 @@ index_partition_cmd: n->subtype = AT_AttachPartition; cmd->name = $3; cmd->bound = NULL; + cmd->concurrent = false; n->def = (Node *) cmd; $$ = (Node *) n; @@ -15276,6 +15291,7 @@ unreserved_keyword: | EXTERNAL | FAMILY | FILTER + | FINALIZE | FIRST_P | FOLLOWING | FORCE @@ -15814,6 +15830,7 @@ bare_label_keyword: | EXTRACT | FALSE_P | FAMILY + | FINALIZE | FIRST_P | FLOAT_P | FOLLOWING diff --git a/src/backend/partitioning/partbounds.c b/src/backend/partitioning/partbounds.c index b9aeb77bc2..336cc1db9b 100644 --- a/src/backend/partitioning/partbounds.c +++ b/src/backend/partitioning/partbounds.c @@ -2808,7 +2808,7 @@ check_new_partition_bound(char *relname, Relation parent, PartitionBoundSpec *spec, ParseState *pstate) { PartitionKey key = RelationGetPartitionKey(parent); - PartitionDesc partdesc = RelationGetPartitionDesc(parent); + PartitionDesc partdesc = RelationGetPartitionDesc(parent, true); PartitionBoundInfo boundinfo = partdesc->boundinfo; int with = -1; bool overlap = false; @@ -4017,7 +4017,7 @@ get_qual_for_list(Relation parent, PartitionBoundSpec *spec) { int i; int ndatums = 0; - PartitionDesc pdesc = RelationGetPartitionDesc(parent); + PartitionDesc pdesc = RelationGetPartitionDesc(parent, true); /* XXX correct? */ PartitionBoundInfo boundinfo = pdesc->boundinfo; if (boundinfo) @@ -4217,7 +4217,7 @@ get_qual_for_range(Relation parent, PartitionBoundSpec *spec, if (spec->is_default) { List *or_expr_args = NIL; - PartitionDesc pdesc = RelationGetPartitionDesc(parent); + PartitionDesc pdesc = RelationGetPartitionDesc(parent, true); /* XXX correct? */ Oid *inhoids = pdesc->oids; int nparts = pdesc->nparts, i; diff --git a/src/backend/partitioning/partdesc.c b/src/backend/partitioning/partdesc.c index f852b6e99d..58570fecfd 100644 --- a/src/backend/partitioning/partdesc.c +++ b/src/backend/partitioning/partdesc.c @@ -37,6 +37,7 @@ typedef struct PartitionDirectoryData { MemoryContext pdir_mcxt; HTAB *pdir_hash; + bool include_detached; } PartitionDirectoryData; typedef struct PartitionDirectoryEntry @@ -46,7 +47,7 @@ typedef struct PartitionDirectoryEntry PartitionDesc pd; } PartitionDirectoryEntry; -static void RelationBuildPartitionDesc(Relation rel); +static void RelationBuildPartitionDesc(Relation rel, bool include_detached); /* @@ -61,13 +62,14 @@ static void RelationBuildPartitionDesc(Relation rel); * that the data doesn't become stale. */ PartitionDesc -RelationGetPartitionDesc(Relation rel) +RelationGetPartitionDesc(Relation rel, bool include_detached) { if (rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) return NULL; - if (unlikely(rel->rd_partdesc == NULL)) - RelationBuildPartitionDesc(rel); + if (unlikely(rel->rd_partdesc == NULL || + rel->rd_partdesc->includes_detached != include_detached)) + RelationBuildPartitionDesc(rel, include_detached); return rel->rd_partdesc; } @@ -88,7 +90,7 @@ RelationGetPartitionDesc(Relation rel) * permanently. */ static void -RelationBuildPartitionDesc(Relation rel) +RelationBuildPartitionDesc(Relation rel, bool include_detached) { PartitionDesc partdesc; PartitionBoundInfo boundinfo = NULL; @@ -110,7 +112,8 @@ RelationBuildPartitionDesc(Relation rel) * concurrently, whatever this function returns will be accurate as of * some well-defined point in time. */ - inhoids = find_inheritance_children(RelationGetRelid(rel), NoLock); + inhoids = find_inheritance_children(RelationGetRelid(rel), include_detached, + NoLock); nparts = list_length(inhoids); /* Allocate working arrays for OIDs, leaf flags, and boundspecs. */ @@ -238,6 +241,7 @@ RelationBuildPartitionDesc(Relation rel) partdesc->boundinfo = partition_bounds_copy(boundinfo, key); partdesc->oids = (Oid *) palloc(nparts * sizeof(Oid)); partdesc->is_leaf = (bool *) palloc(nparts * sizeof(bool)); + partdesc->includes_detached = include_detached; /* * Assign OIDs from the original array into mapped indexes of the @@ -280,7 +284,7 @@ RelationBuildPartitionDesc(Relation rel) * Create a new partition directory object. */ PartitionDirectory -CreatePartitionDirectory(MemoryContext mcxt) +CreatePartitionDirectory(MemoryContext mcxt, bool include_detached) { MemoryContext oldcontext = MemoryContextSwitchTo(mcxt); PartitionDirectory pdir; @@ -295,6 +299,7 @@ CreatePartitionDirectory(MemoryContext mcxt) pdir->pdir_hash = hash_create("partition directory", 256, &ctl, HASH_ELEM | HASH_BLOBS | HASH_CONTEXT); + pdir->include_detached = include_detached; MemoryContextSwitchTo(oldcontext); return pdir; @@ -327,7 +332,7 @@ PartitionDirectoryLookup(PartitionDirectory pdir, Relation rel) */ RelationIncrementReferenceCount(rel); pde->rel = rel; - pde->pd = RelationGetPartitionDesc(rel); + pde->pd = RelationGetPartitionDesc(rel, pdir->include_detached); Assert(pde->pd != NULL); } return pde->pd; diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 53a511f1da..38ff1e0923 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -1274,6 +1274,25 @@ ProcessUtilitySlow(ParseState *pstate, AlterTableStmt *atstmt = (AlterTableStmt *) parsetree; Oid relid; LOCKMODE lockmode; + ListCell *cell; + + /* + * Disallow ALTER TABLE .. DETACH CONCURRENTLY in a + * transaction block or function. (Perhaps it could be + * allowed in a procedure, but don't hold your breath.) + */ + foreach(cell, atstmt->cmds) + { + AlterTableCmd *cmd = (AlterTableCmd *) lfirst(cell); + + /* Disallow DETACH CONCURRENTLY in a transaction block */ + if (cmd->subtype == AT_DetachPartition) + { + if (((PartitionCmd *) cmd->def)->concurrent) + PreventInTransactionBlock(isTopLevel, + "ALTER TABLE ... DETACH CONCURRENTLY"); + } + } /* * Figure out lock mode, and acquire lock. This also does diff --git a/src/backend/utils/cache/partcache.c b/src/backend/utils/cache/partcache.c index a6388d980e..74d85b89fb 100644 --- a/src/backend/utils/cache/partcache.c +++ b/src/backend/utils/cache/partcache.c @@ -341,6 +341,7 @@ generate_partition_qual(Relation rel) bool isnull; List *my_qual = NIL, *result = NIL; + Oid parentrelid; Relation parent; /* Guard against stack overflow due to overly deep partition tree */ @@ -350,9 +351,19 @@ generate_partition_qual(Relation rel) if (rel->rd_partcheckvalid) return copyObject(rel->rd_partcheck); + /* + * Obtain parent relid; if it's invalid, then the partition is being + * detached + */ + parentrelid = get_partition_parent(RelationGetRelid(rel)); + if (parentrelid == InvalidOid) + { + rel->rd_partcheckvalid = true; + return NIL; + } + /* Grab at least an AccessShareLock on the parent table */ - parent = relation_open(get_partition_parent(RelationGetRelid(rel)), - AccessShareLock); + parent = relation_open(parentrelid, AccessShareLock); /* Get pg_class.relpartbound */ tuple = SearchSysCache1(RELOID, RelationGetRelid(rel)); diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index caf97563f4..883ebdc474 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2117,7 +2117,12 @@ describeOneTableDetails(const char *schemaname, printfPQExpBuffer(&buf, "SELECT inhparent::pg_catalog.regclass,\n" - " pg_catalog.pg_get_expr(c.relpartbound, c.oid)"); + " pg_catalog.pg_get_expr(c.relpartbound, c.oid),\n "); + + appendPQExpBuffer(&buf, + pset.sversion >= 140000 ? "inhdetached" : + "false as inhdetached"); + /* If verbose, also request the partition constraint definition */ if (verbose) appendPQExpBufferStr(&buf, @@ -2135,17 +2140,19 @@ describeOneTableDetails(const char *schemaname, { char *parent_name = PQgetvalue(result, 0, 0); char *partdef = PQgetvalue(result, 0, 1); + char *detached = PQgetvalue(result, 0, 2); - printfPQExpBuffer(&tmpbuf, _("Partition of: %s %s"), parent_name, - partdef); + printfPQExpBuffer(&tmpbuf, _("Partition of: %s %s%s"), parent_name, + partdef, + strcmp(detached, "t") == 0 ? " DETACHED" : ""); printTableAddFooter(&cont, tmpbuf.data); if (verbose) { char *partconstraintdef = NULL; - if (!PQgetisnull(result, 0, 2)) - partconstraintdef = PQgetvalue(result, 0, 2); + if (!PQgetisnull(result, 0, 3)) + partconstraintdef = PQgetvalue(result, 0, 3); /* If there isn't any constraint, show that explicitly */ if (partconstraintdef == NULL || partconstraintdef[0] == '\0') printfPQExpBuffer(&tmpbuf, _("No partition constraint")); @@ -3197,9 +3204,18 @@ describeOneTableDetails(const char *schemaname, } /* print child tables (with additional info if partitions) */ - if (pset.sversion >= 100000) + if (pset.sversion >= 140000) printfPQExpBuffer(&buf, - "SELECT c.oid::pg_catalog.regclass, c.relkind," + "SELECT c.oid::pg_catalog.regclass, c.relkind, inhdetached," + " pg_catalog.pg_get_expr(c.relpartbound, c.oid)\n" + "FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\n" + "WHERE c.oid = i.inhrelid AND i.inhparent = '%s'\n" + "ORDER BY pg_catalog.pg_get_expr(c.relpartbound, c.oid) = 'DEFAULT'," + " c.oid::pg_catalog.regclass::pg_catalog.text;", + oid); + else if (pset.sversion >= 100000) + printfPQExpBuffer(&buf, + "SELECT c.oid::pg_catalog.regclass, c.relkind, false AS inhdetached," " pg_catalog.pg_get_expr(c.relpartbound, c.oid)\n" "FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\n" "WHERE c.oid = i.inhrelid AND i.inhparent = '%s'\n" @@ -3208,14 +3224,14 @@ describeOneTableDetails(const char *schemaname, oid); else if (pset.sversion >= 80300) printfPQExpBuffer(&buf, - "SELECT c.oid::pg_catalog.regclass, c.relkind, NULL\n" + "SELECT c.oid::pg_catalog.regclass, c.relkind, false AS inhdetached, NULL\n" "FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\n" "WHERE c.oid = i.inhrelid AND i.inhparent = '%s'\n" "ORDER BY c.oid::pg_catalog.regclass::pg_catalog.text;", oid); else printfPQExpBuffer(&buf, - "SELECT c.oid::pg_catalog.regclass, c.relkind, NULL\n" + "SELECT c.oid::pg_catalog.regclass, c.relkind, false AS inhdetached, NULL\n" "FROM pg_catalog.pg_class c, pg_catalog.pg_inherits i\n" "WHERE c.oid = i.inhrelid AND i.inhparent = '%s'\n" "ORDER BY c.relname;", @@ -3265,11 +3281,13 @@ describeOneTableDetails(const char *schemaname, else printfPQExpBuffer(&buf, "%*s %s", ctw, "", PQgetvalue(result, i, 0)); - if (!PQgetisnull(result, i, 2)) - appendPQExpBuffer(&buf, " %s", PQgetvalue(result, i, 2)); + if (!PQgetisnull(result, i, 3)) + appendPQExpBuffer(&buf, " %s", PQgetvalue(result, i, 3)); if (child_relkind == RELKIND_PARTITIONED_TABLE || child_relkind == RELKIND_PARTITIONED_INDEX) appendPQExpBufferStr(&buf, ", PARTITIONED"); + if (strcmp(PQgetvalue(result, i, 2), "t") == 0) + appendPQExpBuffer(&buf, " (DETACHED)"); if (i < tuples - 1) appendPQExpBufferChar(&buf, ','); diff --git a/src/include/catalog/pg_inherits.h b/src/include/catalog/pg_inherits.h index a0f2ec4e0b..ba2a150128 100644 --- a/src/include/catalog/pg_inherits.h +++ b/src/include/catalog/pg_inherits.h @@ -34,6 +34,7 @@ CATALOG(pg_inherits,2611,InheritsRelationId) Oid inhrelid; Oid inhparent; int32 inhseqno; + bool inhdetached; } FormData_pg_inherits; /* ---------------- @@ -49,7 +50,8 @@ DECLARE_INDEX(pg_inherits_parent_index, 2187, on pg_inherits using btree(inhpare #define InheritsParentIndexId 2187 -extern List *find_inheritance_children(Oid parentrelId, LOCKMODE lockmode); +extern List *find_inheritance_children(Oid parentrelId, bool include_detached, + LOCKMODE lockmode); extern List *find_all_inheritors(Oid parentrelId, LOCKMODE lockmode, List **parents); extern bool has_subclass(Oid relationId); @@ -57,6 +59,7 @@ extern bool has_superclass(Oid relationId); extern bool typeInheritsFrom(Oid subclassTypeId, Oid superclassTypeId); extern void StoreSingleInheritance(Oid relationId, Oid parentOid, int32 seqNumber); -extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent); +extern bool DeleteInheritsTuple(Oid inhrelid, Oid inhparent, bool allow_detached, + const char *childname); #endif /* PG_INHERITS_H */ diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index dc2bb40926..1a2f87f5f9 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -860,6 +860,7 @@ typedef struct PartitionCmd NodeTag type; RangeVar *name; /* name of partition to attach/detach */ PartitionBoundSpec *bound; /* FOR VALUES, if attaching */ + bool concurrent; } PartitionCmd; /**************************************************************************** @@ -1870,6 +1871,7 @@ typedef enum AlterTableType AT_GenericOptions, /* OPTIONS (...) */ AT_AttachPartition, /* ATTACH PARTITION */ AT_DetachPartition, /* DETACH PARTITION */ + AT_DetachPartitionFinalize, /* DETACH PARTITION FINALIZE */ AT_AddIdentity, /* ADD IDENTITY */ AT_SetIdentity, /* SET identity column options */ AT_DropIdentity, /* DROP IDENTITY */ diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 8c554e1f69..c79b76edca 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -163,6 +163,7 @@ PG_KEYWORD("false", FALSE_P, RESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("family", FAMILY, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("fetch", FETCH, RESERVED_KEYWORD, AS_LABEL) PG_KEYWORD("filter", FILTER, UNRESERVED_KEYWORD, AS_LABEL) +PG_KEYWORD("finalize", FINALIZE, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("first", FIRST_P, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("float", FLOAT_P, COL_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("following", FOLLOWING, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/partitioning/partdesc.h b/src/include/partitioning/partdesc.h index ff113199e5..7f03ff4271 100644 --- a/src/include/partitioning/partdesc.h +++ b/src/include/partitioning/partdesc.h @@ -21,6 +21,7 @@ typedef struct PartitionDescData { int nparts; /* Number of partitions */ + bool includes_detached; /* Does it include detached partitions */ Oid *oids; /* Array of 'nparts' elements containing * partition OIDs in order of the their bounds */ bool *is_leaf; /* Array of 'nparts' elements storing whether @@ -30,9 +31,9 @@ typedef struct PartitionDescData } PartitionDescData; -extern PartitionDesc RelationGetPartitionDesc(Relation rel); +extern PartitionDesc RelationGetPartitionDesc(Relation rel, bool include_detached); -extern PartitionDirectory CreatePartitionDirectory(MemoryContext mcxt); +extern PartitionDirectory CreatePartitionDirectory(MemoryContext mcxt, bool include_detached); extern PartitionDesc PartitionDirectoryLookup(PartitionDirectory, Relation); extern void DestroyPartitionDirectory(PartitionDirectory pdir); diff --git a/src/include/utils/snapmgr.h b/src/include/utils/snapmgr.h index 579be352c5..21487b6d44 100644 --- a/src/include/utils/snapmgr.h +++ b/src/include/utils/snapmgr.h @@ -133,6 +133,7 @@ extern void AtEOXact_Snapshot(bool isCommit, bool resetXmin); extern void ImportSnapshot(const char *idstr); extern bool XactHasExportedSnapshots(void); extern void DeleteAllExportedSnapshotFiles(void); +extern void WaitForOlderSnapshots(TransactionId limitXmin, bool progress); extern bool ThereAreNoPriorRegisteredSnapshots(void); extern bool TransactionIdLimitedForOldSnapshots(TransactionId recentXmin, Relation relation, diff --git a/src/test/isolation/expected/detach-partition-concurrently-1.out b/src/test/isolation/expected/detach-partition-concurrently-1.out new file mode 100644 index 0000000000..8d15beac60 --- /dev/null +++ b/src/test/isolation/expected/detach-partition-concurrently-1.out @@ -0,0 +1,301 @@ +Parsed test spec with 3 sessions + +starting permutation: s1b s1s s2detach s1s s1c s1s +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +step s1c: COMMIT; +step s2detach: <... completed> +step s1s: SELECT * FROM d_listp; +a + +1 + +starting permutation: s1b s1s s2detach s1s s3s s3i s1c s3i s2drop s1s +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +step s3s: SELECT * FROM d_listp; +a + +1 +step s3i: SELECT relpartbound IS NULL FROM pg_class where relname = 'd_listp2'; +?column? + +f +step s1c: COMMIT; +step s2detach: <... completed> +step s3i: SELECT relpartbound IS NULL FROM pg_class where relname = 'd_listp2'; +?column? + +t +step s2drop: DROP TABLE d_listp2; +step s1s: SELECT * FROM d_listp; +a + +1 + +starting permutation: s1b s1s s2detach s1ins s1s s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1ins: INSERT INTO d_listp VALUES (1); +step s1s: SELECT * FROM d_listp; +a + +1 +1 +step s1c: COMMIT; +step s2detach: <... completed> + +starting permutation: s1b s1s s1ins2 s2detach s1ins s1s s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1ins2: INSERT INTO d_listp VALUES (2); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1ins: INSERT INTO d_listp VALUES (1); +step s1s: SELECT * FROM d_listp; +a + +1 +1 +2 +2 +step s1c: COMMIT; +step s2detach: <... completed> + +starting permutation: s1brr s1s s2detach s1ins s1s s1c +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1ins: INSERT INTO d_listp VALUES (1); +step s1s: SELECT * FROM d_listp; +a + +1 +1 +2 +step s1c: COMMIT; +step s2detach: <... completed> + +starting permutation: s1brr s1s s2detach s1s s1c +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1c: COMMIT; +step s2detach: <... completed> + +starting permutation: s1b s1ins2 s2detach s3ins2 s1c s3debugtest +step s1b: BEGIN; +step s1ins2: INSERT INTO d_listp VALUES (2); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s3ins2: INSERT INTO d_listp VALUES (2); <waiting ...> +step s1c: COMMIT; +step s2detach: <... completed> +step s3ins2: <... completed> +step s3debugtest: SELECT tableoid::regclass FROM d_listp; SELECT * from d_listp2; +tableoid + +d_listp1 +a + +2 +2 +2 + +starting permutation: s1brr s1prep s1s s2detach s1s s1exec1 s3s s1dealloc s1c +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep: PREPARE f(int) AS INSERT INTO d_listp VALUES ($1); +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1exec1: EXECUTE f(1); +step s3s: SELECT * FROM d_listp; +a + +1 +step s1dealloc: DEALLOCATE f; +step s1c: COMMIT; +step s2detach: <... completed> + +starting permutation: s1brr s1prep s1exec2 s2detach s1s s1exec2 s3s s1c s1dealloc +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep: PREPARE f(int) AS INSERT INTO d_listp VALUES ($1); +step s1exec2: EXECUTE f(2); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +2 +step s1exec2: EXECUTE f(2); +step s3s: SELECT * FROM d_listp; +a + +1 +2 +step s1c: COMMIT; +step s2detach: <... completed> +step s1dealloc: DEALLOCATE f; + +starting permutation: s1brr s1prep s1s s2detach s1s s1exec2 s1c s1dealloc +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep: PREPARE f(int) AS INSERT INTO d_listp VALUES ($1); +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1exec2: EXECUTE f(2); +step s1c: COMMIT; +step s2detach: <... completed> +step s1dealloc: DEALLOCATE f; + +starting permutation: s1brr s1prep s2detach s1s s1exec2 s1c s1dealloc +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep: PREPARE f(int) AS INSERT INTO d_listp VALUES ($1); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1exec2: EXECUTE f(2); +step s1c: COMMIT; +step s2detach: <... completed> +step s1dealloc: DEALLOCATE f; + +starting permutation: s1brr s1prep1 s2detach s1s s1exec2 s1c s1dealloc +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep1: PREPARE f(int) AS INSERT INTO d_listp VALUES (1); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1exec2: EXECUTE f(2); +step s1c: COMMIT; +step s2detach: <... completed> +step s1dealloc: DEALLOCATE f; + +starting permutation: s1brr s1prep2 s2detach s1s s1exec2 s1c s1dealloc +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1prep2: PREPARE f(int) AS INSERT INTO d_listp VALUES (2); +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s1exec2: EXECUTE f(2); +step s1c: COMMIT; +step s2detach: <... completed> +step s1dealloc: DEALLOCATE f; + +starting permutation: s1b s1s s2detach s3drop1 s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s3drop1: DROP TABLE d_listp; <waiting ...> +step s1c: COMMIT; +step s2detach: <... completed> +step s3drop1: <... completed> +error in steps s1c s2detach s3drop1: ERROR: partitioned table "d_listp" was removed concurrently + +starting permutation: s1b s1s s2detach s3drop2 s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s3drop2: DROP TABLE d_listp2; <waiting ...> +step s1c: COMMIT; +step s2detach: <... completed> +step s3drop2: <... completed> +error in steps s1c s2detach s3drop2: ERROR: partition "d_listp2" was removed concurrently + +starting permutation: s1b s1s s2detach s3detach s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s3detach: ALTER TABLE d_listp DETACH PARTITION d_listp2; <waiting ...> +step s1c: COMMIT; +step s2detach: <... completed> +step s3detach: <... completed> +error in steps s1c s2detach s3detach: ERROR: cannot detach partition "d_listp2" + +starting permutation: s1b s1s s2detach s3rename s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_listp; +a + +1 +2 +step s2detach: ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; <waiting ...> +step s3rename: ALTER TABLE d_listp2 RENAME TO d_listp_foobar; <waiting ...> +step s1c: COMMIT; +step s2detach: <... completed> +step s3rename: <... completed> diff --git a/src/test/isolation/expected/detach-partition-concurrently-2.out b/src/test/isolation/expected/detach-partition-concurrently-2.out new file mode 100644 index 0000000000..85be707b40 --- /dev/null +++ b/src/test/isolation/expected/detach-partition-concurrently-2.out @@ -0,0 +1,66 @@ +Parsed test spec with 3 sessions + +starting permutation: s1b s1s s2d s3i1 s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_lp_fk; +a + +1 +2 +step s2d: ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; <waiting ...> +step s3i1: INSERT INTO d_lp_fk_r VALUES (1); +ERROR: insert or update on table "d_lp_fk_r" violates foreign key constraint "d_lp_fk_r_a_fkey" +step s1c: COMMIT; +step s2d: <... completed> + +starting permutation: s1b s1s s2d s3i2 s3i2 s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_lp_fk; +a + +1 +2 +step s2d: ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; <waiting ...> +step s3i2: INSERT INTO d_lp_fk_r VALUES (2); +step s3i2: INSERT INTO d_lp_fk_r VALUES (2); +step s1c: COMMIT; +step s2d: <... completed> + +starting permutation: s1b s1s s3i1 s2d s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_lp_fk; +a + +1 +2 +step s3i1: INSERT INTO d_lp_fk_r VALUES (1); +step s2d: ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; +ERROR: removing partition "d_lp_fk_1" violates foreign key constraint "d_lp_fk_r_a_fkey1" +step s1c: COMMIT; + +starting permutation: s1b s1s s3i2 s2d s1c +step s1b: BEGIN; +step s1s: SELECT * FROM d_lp_fk; +a + +1 +2 +step s3i2: INSERT INTO d_lp_fk_r VALUES (2); +step s2d: ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; <waiting ...> +step s1c: COMMIT; +step s2d: <... completed> + +starting permutation: s1b s1s s3b s2d s3i1 s1c s3c +step s1b: BEGIN; +step s1s: SELECT * FROM d_lp_fk; +a + +1 +2 +step s3b: BEGIN; +step s2d: ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; <waiting ...> +step s3i1: INSERT INTO d_lp_fk_r VALUES (1); +ERROR: insert or update on table "d_lp_fk_r" violates foreign key constraint "d_lp_fk_r_a_fkey" +step s1c: COMMIT; +step s2d: <... completed> +step s3c: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index f2e752c445..3c8fa4e07d 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -24,6 +24,8 @@ test: deadlock-hard test: deadlock-soft test: deadlock-soft-2 test: deadlock-parallel +test: detach-partition-concurrently-1 +test: detach-partition-concurrently-2 test: fk-contention test: fk-deadlock test: fk-deadlock2 diff --git a/src/test/isolation/specs/detach-partition-concurrently-1.spec b/src/test/isolation/specs/detach-partition-concurrently-1.spec new file mode 100644 index 0000000000..1e9bea8198 --- /dev/null +++ b/src/test/isolation/specs/detach-partition-concurrently-1.spec @@ -0,0 +1,90 @@ +# Test that detach partition concurrently makes the partition invisible at the +# correct time. + +setup +{ + DROP TABLE IF EXISTS d_listp, d_listp1, d_listp2; + CREATE TABLE d_listp (a int) PARTITION BY LIST(a); + CREATE TABLE d_listp1 PARTITION OF d_listp FOR VALUES IN (1); + CREATE TABLE d_listp2 PARTITION OF d_listp FOR VALUES IN (2); + INSERT INTO d_listp VALUES (1),(2); +} + +teardown { + DROP TABLE IF EXISTS d_listp, d_listp2, d_listp_foobar; +} + +session "s1" +step "s1b" { BEGIN; } +step "s1brr" { BEGIN ISOLATION LEVEL REPEATABLE READ; } +step "s1s" { SELECT * FROM d_listp; } +step "s1ins" { INSERT INTO d_listp VALUES (1); } +step "s1ins2" { INSERT INTO d_listp VALUES (2); } +step "s1prep" { PREPARE f(int) AS INSERT INTO d_listp VALUES ($1); } +step "s1prep1" { PREPARE f(int) AS INSERT INTO d_listp VALUES (1); } +step "s1prep2" { PREPARE f(int) AS INSERT INTO d_listp VALUES (2); } +step "s1exec1" { EXECUTE f(1); } +step "s1exec2" { EXECUTE f(2); } +step "s1dealloc" { DEALLOCATE f; } +step "s1c" { COMMIT; } + +session "s2" +step "s2detach" { ALTER TABLE d_listp DETACH PARTITION d_listp2 CONCURRENTLY; } +step "s2drop" { DROP TABLE d_listp2; } + +session "s3" +step "s3s" { SELECT * FROM d_listp; } +step "s3i" { SELECT relpartbound IS NULL FROM pg_class where relname = 'd_listp2'; } +step "s3ins2" { INSERT INTO d_listp VALUES (2); } +step "s3drop1" { DROP TABLE d_listp; } +step "s3drop2" { DROP TABLE d_listp2; } +step "s3detach" { ALTER TABLE d_listp DETACH PARTITION d_listp2; } +step "s3rename" { ALTER TABLE d_listp2 RENAME TO d_listp_foobar; } +# XXX remove this step +step "s3debugtest" { SELECT tableoid::regclass FROM d_listp; SELECT * from d_listp2; } + +# The transaction that detaches hangs until it sees any older transaction +# terminate, as does anybody else. +permutation "s1b" "s1s" "s2detach" "s1s" "s1c" "s1s" + +# relpartbound remains set until s1 commits +# XXX this could be timing dependent :-( +permutation "s1b" "s1s" "s2detach" "s1s" "s3s" "s3i" "s1c" "s3i" "s2drop" "s1s" + +# In read-committed mode, the partition disappears from view of concurrent +# transactions immediately. But if a write lock is held, then the detach +# has to wait. +permutation "s1b" "s1s" "s2detach" "s1ins" "s1s" "s1c" +permutation "s1b" "s1s" "s1ins2" "s2detach" "s1ins" "s1s" "s1c" + +# In repeatable-read mode, the partition remains visible until commit even +# if the to-be-detached partition is not locked for write. +permutation "s1brr" "s1s" "s2detach" "s1ins" "s1s" "s1c" +permutation "s1brr" "s1s" "s2detach" "s1s" "s1c" + +# Another process trying to acquire a write lock will be blocked behind the +# detacher +permutation "s1b" "s1ins2" "s2detach" "s3ins2" "s1c" "s3debugtest" + +# a prepared query is not blocked +permutation "s1brr" "s1prep" "s1s" "s2detach" "s1s" "s1exec1" "s3s" "s1dealloc" "s1c" +permutation "s1brr" "s1prep" "s1exec2" "s2detach" "s1s" "s1exec2" "s3s" "s1c" "s1dealloc" +permutation "s1brr" "s1prep" "s1s" "s2detach" "s1s" "s1exec2" "s1c" "s1dealloc" +permutation "s1brr" "s1prep" "s2detach" "s1s" "s1exec2" "s1c" "s1dealloc" +permutation "s1brr" "s1prep1" "s2detach" "s1s" "s1exec2" "s1c" "s1dealloc" +permutation "s1brr" "s1prep2" "s2detach" "s1s" "s1exec2" "s1c" "s1dealloc" + + +# Note: the following tests are not very interesting in isolationtester because +# the way this tool handles multiple sessions: it'll only run the drop/detach/ +# rename commands after s2 is blocked waiting. It would be more useful to +# run the DDL when s2 commits its first transaction instead. Maybe these should +# become TAP tests someday. + +# What happens if the partition or the parent table is dropped while waiting? +permutation "s1b" "s1s" "s2detach" "s3drop1" "s1c" +permutation "s1b" "s1s" "s2detach" "s3drop2" "s1c" +# Also try a non-concurrent detach while the other one is waiting +permutation "s1b" "s1s" "s2detach" "s3detach" "s1c" +# and some other DDL +permutation "s1b" "s1s" "s2detach" "s3rename" "s1c" diff --git a/src/test/isolation/specs/detach-partition-concurrently-2.spec b/src/test/isolation/specs/detach-partition-concurrently-2.spec new file mode 100644 index 0000000000..9281c80a69 --- /dev/null +++ b/src/test/isolation/specs/detach-partition-concurrently-2.spec @@ -0,0 +1,41 @@ +# Test that detach partition concurrently makes the partition safe +# for foreign keys that reference it. + +setup +{ + DROP TABLE IF EXISTS d_lp_fk, d_lp_fk_1, d_lp_fk_2, d_lp_fk_r; + + CREATE TABLE d_lp_fk (a int PRIMARY KEY) PARTITION BY LIST(a); + CREATE TABLE d_lp_fk_1 PARTITION OF d_lp_fk FOR VALUES IN (1); + CREATE TABLE d_lp_fk_2 PARTITION OF d_lp_fk FOR VALUES IN (2); + INSERT INTO d_lp_fk VALUES (1), (2); + + CREATE TABLE d_lp_fk_r (a int references d_lp_fk); +} + +teardown { DROP TABLE IF EXISTS d_lp_fk, d_lp_fk_1, d_lp_fk_2, d_lp_fk_r; } + +session "s1" +step "s1b" { BEGIN; } +step "s1s" { SELECT * FROM d_lp_fk; } +step "s1c" { COMMIT; } + +session "s2" +step "s2d" { ALTER TABLE d_lp_fk DETACH PARTITION d_lp_fk_1 CONCURRENTLY; } + +session "s3" +step "s3b" { BEGIN; } +step "s3i1" { INSERT INTO d_lp_fk_r VALUES (1); } +step "s3i2" { INSERT INTO d_lp_fk_r VALUES (2); } +step "s3c" { COMMIT; } + +# The transaction that detaches hangs until it sees any older transaction +# terminate. +permutation "s1b" "s1s" "s2d" "s3i1" "s1c" +permutation "s1b" "s1s" "s2d" "s3i2" "s3i2" "s1c" + +permutation "s1b" "s1s" "s3i1" "s2d" "s1c" +permutation "s1b" "s1s" "s3i2" "s2d" "s1c" + +# what if s3 has an uncommitted insertion? +permutation "s1b" "s1s" "s3b" "s2d" "s3i1" "s1c" "s3c" diff --git a/src/test/modules/delay_execution/Makefile b/src/test/modules/delay_execution/Makefile index f270aebf3a..70f24e846d 100644 --- a/src/test/modules/delay_execution/Makefile +++ b/src/test/modules/delay_execution/Makefile @@ -7,7 +7,8 @@ OBJS = \ $(WIN32RES) \ delay_execution.o -ISOLATION = partition-addition +ISOLATION = partition-addition \ + partition-removal-1 ifdef USE_PGXS PG_CONFIG = pg_config diff --git a/src/test/modules/delay_execution/expected/partition-removal-1.out b/src/test/modules/delay_execution/expected/partition-removal-1.out new file mode 100644 index 0000000000..00b93ecf65 --- /dev/null +++ b/src/test/modules/delay_execution/expected/partition-removal-1.out @@ -0,0 +1,79 @@ +Parsed test spec with 3 sessions + +starting permutation: s3lock s1b s1exec s2remp s3unlock s1c +step s3lock: SELECT pg_advisory_lock(12543); +pg_advisory_lock + + +step s1b: BEGIN; +step s1exec: SELECT * FROM partrem WHERE a <> 1 AND a <> (SELECT 3); <waiting ...> +step s2remp: ALTER TABLE partrem DETACH PARTITION partrem2 CONCURRENTLY; <waiting ...> +step s3unlock: SELECT pg_advisory_unlock(12543); +pg_advisory_unlock + +t +step s1exec: <... completed> +a b + +2 JKL +step s1c: COMMIT; +step s2remp: <... completed> + +starting permutation: s3lock s1brr s1exec s2remp s3unlock s1c +step s3lock: SELECT pg_advisory_lock(12543); +pg_advisory_lock + + +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1exec: SELECT * FROM partrem WHERE a <> 1 AND a <> (SELECT 3); <waiting ...> +step s2remp: ALTER TABLE partrem DETACH PARTITION partrem2 CONCURRENTLY; <waiting ...> +step s3unlock: SELECT pg_advisory_unlock(12543); +pg_advisory_unlock + +t +step s1exec: <... completed> +a b + +2 JKL +step s1c: COMMIT; +step s2remp: <... completed> + +starting permutation: s3lock s1b s1exec2 s2remp s3unlock s1c +step s3lock: SELECT pg_advisory_lock(12543); +pg_advisory_lock + + +step s1b: BEGIN; +step s1exec2: SELECT * FROM partrem WHERE a <> (SELECT 2); <waiting ...> +step s2remp: ALTER TABLE partrem DETACH PARTITION partrem2 CONCURRENTLY; <waiting ...> +step s3unlock: SELECT pg_advisory_unlock(12543); +pg_advisory_unlock + +t +step s1exec2: <... completed> +a b + +1 ABC +3 DEF +step s1c: COMMIT; +step s2remp: <... completed> + +starting permutation: s3lock s1brr s1exec2 s2remp s3unlock s1c +step s3lock: SELECT pg_advisory_lock(12543); +pg_advisory_lock + + +step s1brr: BEGIN ISOLATION LEVEL REPEATABLE READ; +step s1exec2: SELECT * FROM partrem WHERE a <> (SELECT 2); <waiting ...> +step s2remp: ALTER TABLE partrem DETACH PARTITION partrem2 CONCURRENTLY; <waiting ...> +step s3unlock: SELECT pg_advisory_unlock(12543); +pg_advisory_unlock + +t +step s1exec2: <... completed> +a b + +1 ABC +3 DEF +step s1c: COMMIT; +step s2remp: <... completed> diff --git a/src/test/modules/delay_execution/specs/partition-removal-1.spec b/src/test/modules/delay_execution/specs/partition-removal-1.spec new file mode 100644 index 0000000000..e8e99f15bf --- /dev/null +++ b/src/test/modules/delay_execution/specs/partition-removal-1.spec @@ -0,0 +1,45 @@ +# Test removal of a partition with less-than-exclusive locking. + +setup +{ + CREATE TABLE partrem (a int, b text) PARTITION BY LIST(a); + CREATE TABLE partrem1 PARTITION OF partrem FOR VALUES IN (1); + CREATE TABLE partrem2 PARTITION OF partrem FOR VALUES IN (2); + CREATE TABLE partrem3 PARTITION OF partrem FOR VALUES IN (3); + INSERT INTO partrem VALUES (1, 'ABC'); + INSERT INTO partrem VALUES (2, 'JKL'); + INSERT INTO partrem VALUES (3, 'DEF'); +} + +teardown +{ + DROP TABLE IF EXISTS partrem, partrem2; +} + +# The SELECT will be planned with all three partitions shown above, +# of which we expect partrem1 to be pruned at planning and partrem3 at +# execution. Then we'll block, and by the time the query is actually +# executed, detach of partrem2 is already underway; however we expect +# its rows to still appear in the result. + +session "s1" +setup { LOAD 'delay_execution'; + SET delay_execution.post_planning_lock_id = 12543; } +step "s1b" { BEGIN; } +step "s1brr" { BEGIN ISOLATION LEVEL REPEATABLE READ; } +step "s1exec" { SELECT * FROM partrem WHERE a <> 1 AND a <> (SELECT 3); } +step "s1exec2" { SELECT * FROM partrem WHERE a <> (SELECT 2); } +step "s1c" { COMMIT; } + +session "s2" +step "s2remp" { ALTER TABLE partrem DETACH PARTITION partrem2 CONCURRENTLY; } + +session "s3" +step "s3lock" { SELECT pg_advisory_lock(12543); } +step "s3unlock" { SELECT pg_advisory_unlock(12543); } + +permutation "s3lock" "s1b" "s1exec" "s2remp" "s3unlock" "s1c" +permutation "s3lock" "s1brr" "s1exec" "s2remp" "s3unlock" "s1c" + +permutation "s3lock" "s1b" "s1exec2" "s2remp" "s3unlock" "s1c" +permutation "s3lock" "s1brr" "s1exec2" "s2remp" "s3unlock" "s1c" -- 2.20.1 --/04w6evG8XlLl3ft-- ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v12 14/17] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 262 +++++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 341 ++-------------------- src/include/access/heapam.h | 17 ++ src/include/access/relscan.h | 8 + src/include/access/tableam.h | 26 +- src/include/nodes/execnodes.h | 14 - 7 files changed, 355 insertions(+), 339 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 27e9d05f14b..086dae59668 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -948,8 +948,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1032,6 +1040,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1044,6 +1058,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1069,6 +1089,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd630e38fa8..36c6fb021c3 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -61,6 +61,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2197,6 +2200,73 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2215,6 +2285,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2355,6 +2427,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->rs_pf_bhs_iterator && + hscan->pfblockno < hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2365,6 +2449,154 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + + if (pstate == NULL) + { + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + bhs_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + bhs_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2390,6 +2622,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index d61965a2761..187b288e688 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); static BitmapHeapIterator *bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, @@ -122,7 +118,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -142,7 +137,7 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) @@ -154,7 +149,16 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; - if (!pstate || init_shared_state) + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif + + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); @@ -169,23 +173,16 @@ BitmapHeapNext(BitmapHeapScanState *node) * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); + BitmapDoneInitializingSharedState(node->pstate); } } @@ -216,19 +213,19 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + scan->rs_bhs_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->tbmiterator : InvalidDsaPointer, + scan->bm_parallel ? scan->bm_parallel->tbmiterator : InvalidDsaPointer, dsa); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { - node->pf_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->prefetch_iterator : InvalidDsaPointer, - dsa); - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; + scan->rs_pf_bhs_iterator = bhs_begin_iterate(tbm, + scan->bm_parallel ? scan->bm_parallel->prefetch_iterator : InvalidDsaPointer, + dsa); } #endif /* USE_PREFETCH */ @@ -243,36 +240,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -296,23 +263,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->pf_iterator && - node->pfblockno < node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -336,219 +289,6 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - - if (pstate == NULL) - { - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - bhs_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - bhs_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} - /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual */ @@ -594,22 +334,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) if (node->ss.ss_currentScanDesc) table_rescan(node->ss.ss_currentScanDesc, NULL); - /* release bitmaps and buffers if any */ - if (node->pf_iterator) - { - bhs_end_iterate(node->pf_iterator); - node->pf_iterator = NULL; - } + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -648,14 +378,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (node->pf_iterator) - bhs_end_iterate(node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -688,17 +414,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->pf_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -738,13 +458,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -828,7 +541,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 49f212eb9a6..accbc1749cd 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -82,6 +82,23 @@ typedef struct HeapScanDescData Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* + * These fields only used for prefetching in bitmap table scans + */ + + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used in serial BHS + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; + /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ int rs_ntuples; /* number of visible tuples on page */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index fb22f305bf6..7938b741d66 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct BitmapHeapIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -45,6 +46,13 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct BitmapHeapIterator *rs_bhs_iterator; + struct BitmapHeapIterator *rs_pf_bhs_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + /* used to validate BHS prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index a06ed271eb6..a30c0412901 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -807,17 +807,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -967,6 +956,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->rs_bhs_iterator = NULL; + result->rs_pf_bhs_iterator = NULL; + result->prefetch_maximum = 0; + result->bm_parallel = NULL; return result; } @@ -1018,6 +1010,12 @@ table_endscan(TableScanDesc scan) { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1034,6 +1032,12 @@ table_rescan(TableScanDesc scan, { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index cf8b4995f0d..592215d5ee7 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1794,18 +1794,11 @@ struct BitmapHeapIterator; * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1813,18 +1806,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - struct BitmapHeapIterator *pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --6jpz2j246qmht4bt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v12-0015-Remove-table_scan_bitmap_next_block.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v8 13/17] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 268 +++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 397 +++------------------- src/include/access/heapam.h | 12 + src/include/access/relscan.h | 11 + src/include/access/tableam.h | 38 ++- src/include/nodes/execnodes.h | 16 - 7 files changed, 388 insertions(+), 380 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index e7bed84f75..c12563a188 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -951,8 +951,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1035,6 +1043,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1047,6 +1061,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1072,6 +1092,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index adfc77684a..efd2784e03 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -55,6 +55,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2112,6 +2115,76 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + TBMIterator *prefetch_iterator = scan->rs_base.pf_tbmiterator; + + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + tbm_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + TBMSharedIterator *prefetch_iterator = scan->rs_base.pf_shared_tbmiterator; + + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + tbm_shared_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2130,6 +2203,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2273,6 +2348,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->pf_tbmiterator && + hscan->pfblockno > hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2283,6 +2370,157 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + + if (pstate == NULL) + { + TBMIterator *prefetch_iterator = scan->rs_base.pf_tbmiterator; + + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + tbm_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + tbm_end_iterate(prefetch_iterator); + scan->rs_base.pf_tbmiterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + TBMSharedIterator *prefetch_iterator = scan->rs_base.pf_shared_tbmiterator; + + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + tbm_shared_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + tbm_end_shared_iterate(prefetch_iterator); + scan->rs_base.pf_shared_tbmiterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2308,6 +2546,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 51c4360205..f241f4cb2c 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); @@ -71,7 +67,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -91,83 +86,53 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) { - TBMIterator *tbmiterator = NULL; - TBMSharedIterator *shared_tbmiterator = NULL; + /* + * The leader will immediately come out of the function, but others + * will be blocked until leader populates the TBM and wakes them up. + */ + bool init_shared_state = node->pstate ? + BitmapShouldInitializeSharedState(node->pstate) : false; + + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif - if (!pstate) + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); if (!tbm || !IsA(tbm, TIDBitmap)) elog(ERROR, "unrecognized result from subplan"); - node->tbm = tbm; - tbmiterator = tbm_begin_iterate(tbm); -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (init_shared_state) { - node->prefetch_iterator = tbm_begin_iterate(tbm); - node->prefetch_pages = 0; - node->prefetch_target = -1; - } -#endif /* USE_PREFETCH */ - } - else - { - /* - * The leader will immediately come out of the function, but - * others will be blocked until leader populates the TBM and wakes - * them up. - */ - if (BitmapShouldInitializeSharedState(pstate)) - { - tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); - if (!tbm || !IsA(tbm, TIDBitmap)) - elog(ERROR, "unrecognized result from subplan"); - - node->tbm = tbm; - /* * Prepare to iterate over the TBM. This will return the * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif - /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); - } - - /* Allocate a private iterator and attach the shared state to it */ - shared_tbmiterator = tbm_attach_shared_iterate(dsa, pstate->tbmiterator); - -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) - { - node->shared_prefetch_iterator = - tbm_attach_shared_iterate(dsa, pstate->prefetch_iterator); + BitmapDoneInitializingSharedState(node->pstate); } -#endif /* USE_PREFETCH */ } /* @@ -197,8 +162,26 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } - scan->tbmiterator = tbmiterator; - scan->shared_tbmiterator = shared_tbmiterator; + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + + if (!scan->bm_parallel) + scan->tbmiterator = tbm_begin_iterate(tbm); + else + /* Allocate a private iterator and attach the shared state to it */ + scan->shared_tbmiterator = tbm_attach_shared_iterate(dsa, scan->bm_parallel->tbmiterator); + +#ifdef USE_PREFETCH + if (scan->prefetch_maximum > 0) + { + if (!scan->bm_parallel) + scan->pf_tbmiterator = tbm_begin_iterate(tbm); + else + scan->pf_shared_tbmiterator = + tbm_attach_shared_iterate(dsa, scan->bm_parallel->prefetch_iterator); + } +#endif /* USE_PREFETCH */ + node->initialized = true; @@ -211,36 +194,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -264,23 +217,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->prefetch_iterator && - node->pfblockno > node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -304,224 +243,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - TBMIterator *prefetch_iterator = node->prefetch_iterator; - - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - tbm_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - TBMSharedIterator *prefetch_iterator = node->shared_prefetch_iterator; - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - tbm_shared_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - TBMIterator *prefetch_iterator = node->prefetch_iterator; - - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - tbm_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - tbm_end_iterate(prefetch_iterator); - node->prefetch_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - TBMSharedIterator *prefetch_iterator = node->shared_prefetch_iterator; - - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - tbm_shared_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - tbm_end_shared_iterate(prefetch_iterator); - node->shared_prefetch_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual @@ -569,22 +291,11 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) table_rescan(node->ss.ss_currentScanDesc, NULL); /* release bitmaps and buffers if any */ - if (node->prefetch_iterator) - tbm_end_iterate(node->prefetch_iterator); - if (node->shared_prefetch_iterator) - tbm_end_shared_iterate(node->shared_prefetch_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; - node->prefetch_iterator = NULL; node->initialized = false; - node->shared_prefetch_iterator = NULL; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -625,14 +336,8 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) /* * release bitmaps and buffers if any */ - if (node->prefetch_iterator) - tbm_end_iterate(node->prefetch_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->shared_prefetch_iterator) - tbm_end_shared_iterate(node->shared_prefetch_iterator); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -665,18 +370,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->prefetch_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; - scanstate->shared_prefetch_iterator = NULL; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -716,13 +414,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -806,7 +497,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 3dfb19ec7d..22bdccc2a9 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -81,6 +81,18 @@ typedef struct HeapScanDescData */ Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used for prefetching in bitmap table scans + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index 92b829cebc..93168bd350 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct TBMIterator; struct TBMSharedIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -46,6 +47,16 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct TBMIterator *tbmiterator; struct TBMSharedIterator *shared_tbmiterator; + /* Prefetch iterators */ + struct TBMIterator *pf_tbmiterator; + struct TBMSharedIterator *pf_shared_tbmiterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + + /* used to validate prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 1d4b79a73f..9cab4462d6 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -800,17 +800,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -961,6 +950,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->shared_tbmiterator = NULL; result->tbmiterator = NULL; + result->pf_shared_tbmiterator = NULL; + result->pf_tbmiterator = NULL; + result->bm_parallel = NULL; return result; } @@ -1029,11 +1021,23 @@ table_endscan(TableScanDesc scan) scan->shared_tbmiterator = NULL; } + if (scan->pf_shared_tbmiterator) + { + tbm_end_shared_iterate(scan->pf_shared_tbmiterator); + scan->pf_shared_tbmiterator = NULL; + } + if (scan->tbmiterator) { tbm_end_iterate(scan->tbmiterator); scan->tbmiterator = NULL; } + + if (scan->pf_tbmiterator) + { + tbm_end_iterate(scan->pf_tbmiterator); + scan->pf_tbmiterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1054,11 +1058,23 @@ table_rescan(TableScanDesc scan, scan->shared_tbmiterator = NULL; } + if (scan->pf_shared_tbmiterator) + { + tbm_end_shared_iterate(scan->pf_shared_tbmiterator); + scan->pf_shared_tbmiterator = NULL; + } + if (scan->tbmiterator) { tbm_end_iterate(scan->tbmiterator); scan->tbmiterator = NULL; } + + if (scan->pf_tbmiterator) + { + tbm_end_iterate(scan->pf_tbmiterator); + scan->pf_tbmiterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 8688bc5ab0..7a3fdf9cd4 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1783,19 +1783,11 @@ typedef struct ParallelBitmapHeapState * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * prefetch_iterator iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate - * shared_prefetch_iterator shared iterator for prefetching * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1803,19 +1795,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - TBMIterator *prefetch_iterator; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - TBMSharedIterator *shared_prefetch_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --xqq4defy3uncu6k6 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0014-Unify-parallel-and-serial-BitmapHeapScan-iterator.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v11 14/17] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 262 +++++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 341 ++-------------------- src/include/access/heapam.h | 17 ++ src/include/access/relscan.h | 8 + src/include/access/tableam.h | 26 +- src/include/nodes/execnodes.h | 14 - 7 files changed, 355 insertions(+), 339 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index ed3a3607b7..614d715fc7 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -948,8 +948,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1032,6 +1040,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1044,6 +1058,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1069,6 +1089,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index f7e4d1094d..68bbb6f88c 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -61,6 +61,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2170,6 +2173,73 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2188,6 +2258,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2328,6 +2400,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->rs_pf_bhs_iterator && + hscan->pfblockno > hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2338,6 +2422,154 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + + if (pstate == NULL) + { + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + bhs_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + bhs_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2363,6 +2595,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 78f79aafff..187b288e68 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); static BitmapHeapIterator *bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, @@ -122,7 +118,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -142,7 +137,7 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) @@ -154,7 +149,16 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; - if (!pstate || init_shared_state) + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif + + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); @@ -169,23 +173,16 @@ BitmapHeapNext(BitmapHeapScanState *node) * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); + BitmapDoneInitializingSharedState(node->pstate); } } @@ -216,19 +213,19 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + scan->rs_bhs_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->tbmiterator : InvalidDsaPointer, + scan->bm_parallel ? scan->bm_parallel->tbmiterator : InvalidDsaPointer, dsa); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { - node->pf_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->prefetch_iterator : InvalidDsaPointer, - dsa); - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; + scan->rs_pf_bhs_iterator = bhs_begin_iterate(tbm, + scan->bm_parallel ? scan->bm_parallel->prefetch_iterator : InvalidDsaPointer, + dsa); } #endif /* USE_PREFETCH */ @@ -243,36 +240,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -296,23 +263,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->pf_iterator && - node->pfblockno > node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -336,219 +289,6 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - - if (pstate == NULL) - { - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - bhs_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - bhs_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} - /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual */ @@ -594,22 +334,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) if (node->ss.ss_currentScanDesc) table_rescan(node->ss.ss_currentScanDesc, NULL); - /* release bitmaps and buffers if any */ - if (node->pf_iterator) - { - bhs_end_iterate(node->pf_iterator); - node->pf_iterator = NULL; - } + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -648,14 +378,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (node->pf_iterator) - bhs_end_iterate(node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -688,17 +414,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->pf_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -738,13 +458,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -828,7 +541,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index c7a538221a..4726d31403 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -82,6 +82,23 @@ typedef struct HeapScanDescData Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* + * These fields only used for prefetching in bitmap table scans + */ + + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used in serial BHS + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; + /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ int rs_ntuples; /* number of visible tuples on page */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index fb22f305bf..7938b741d6 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct BitmapHeapIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -45,6 +46,13 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct BitmapHeapIterator *rs_bhs_iterator; + struct BitmapHeapIterator *rs_pf_bhs_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + /* used to validate BHS prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index bb2b79717c..799ac013d4 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -811,17 +811,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -971,6 +960,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->rs_bhs_iterator = NULL; + result->rs_pf_bhs_iterator = NULL; + result->prefetch_maximum = 0; + result->bm_parallel = NULL; return result; } @@ -1035,6 +1027,12 @@ table_endscan(TableScanDesc scan) { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1051,6 +1049,12 @@ table_rescan(TableScanDesc scan, { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 52cedd1b35..60916bf0d0 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1785,18 +1785,11 @@ struct BitmapHeapIterator; * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1804,18 +1797,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - struct BitmapHeapIterator *pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --owzzsiozz6hgpp7e Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0015-Remove-table_scan_bitmap_next_block.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v13 14/16] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 262 +++++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 341 ++-------------------- src/include/access/heapam.h | 17 ++ src/include/access/relscan.h | 8 + src/include/access/tableam.h | 26 +- src/include/nodes/execnodes.h | 14 - 7 files changed, 355 insertions(+), 339 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 50b4bc1475..263c728543 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -948,8 +948,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1032,6 +1040,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1044,6 +1058,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1069,6 +1089,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index d9ceb4b848..42d5b749de 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -60,6 +60,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2186,6 +2189,73 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2204,6 +2274,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2344,6 +2416,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->rs_pf_bhs_iterator && + hscan->pfblockno < hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2354,6 +2438,154 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + + if (pstate == NULL) + { + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + bhs_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + bhs_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2379,6 +2611,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index d61965a276..187b288e68 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); static BitmapHeapIterator *bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, @@ -122,7 +118,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -142,7 +137,7 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) @@ -154,7 +149,16 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; - if (!pstate || init_shared_state) + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif + + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); @@ -169,23 +173,16 @@ BitmapHeapNext(BitmapHeapScanState *node) * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); + BitmapDoneInitializingSharedState(node->pstate); } } @@ -216,19 +213,19 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + scan->rs_bhs_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->tbmiterator : InvalidDsaPointer, + scan->bm_parallel ? scan->bm_parallel->tbmiterator : InvalidDsaPointer, dsa); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { - node->pf_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->prefetch_iterator : InvalidDsaPointer, - dsa); - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; + scan->rs_pf_bhs_iterator = bhs_begin_iterate(tbm, + scan->bm_parallel ? scan->bm_parallel->prefetch_iterator : InvalidDsaPointer, + dsa); } #endif /* USE_PREFETCH */ @@ -243,36 +240,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -296,23 +263,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->pf_iterator && - node->pfblockno < node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -336,219 +289,6 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - - if (pstate == NULL) - { - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - bhs_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - bhs_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} - /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual */ @@ -594,22 +334,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) if (node->ss.ss_currentScanDesc) table_rescan(node->ss.ss_currentScanDesc, NULL); - /* release bitmaps and buffers if any */ - if (node->pf_iterator) - { - bhs_end_iterate(node->pf_iterator); - node->pf_iterator = NULL; - } + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -648,14 +378,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (node->pf_iterator) - bhs_end_iterate(node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -688,17 +414,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->pf_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -738,13 +458,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -828,7 +541,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index a527e1b99f..bf212f577f 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -86,6 +86,23 @@ typedef struct HeapScanDescData Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* + * These fields only used for prefetching in bitmap table scans + */ + + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used in serial BHS + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; + /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ int rs_ntuples; /* number of visible tuples on page */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index fb22f305bf..7938b741d6 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct BitmapHeapIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -45,6 +46,13 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct BitmapHeapIterator *rs_bhs_iterator; + struct BitmapHeapIterator *rs_pf_bhs_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + /* used to validate BHS prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 70e538b76b..a2b229fb87 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -785,17 +785,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -945,6 +934,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->rs_bhs_iterator = NULL; + result->rs_pf_bhs_iterator = NULL; + result->prefetch_maximum = 0; + result->bm_parallel = NULL; return result; } @@ -996,6 +988,12 @@ table_endscan(TableScanDesc scan) { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1012,6 +1010,12 @@ table_rescan(TableScanDesc scan, { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index cf8b4995f0..592215d5ee 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1794,18 +1794,11 @@ struct BitmapHeapIterator; * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1813,18 +1806,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - struct BitmapHeapIterator *pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --cuuqjeyokkhgd736 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v13-0015-Remove-table_scan_bitmap_next_block.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v15 11/13] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) 7566751f0ce9 eliminated a table AM layering violation for the current block in a BitmapHeapScan but left the violation in BitmapHeapScan prefetching code. To resolve this, move the BitmapHeapScan prefetching logic entirely into heap AM code. There is a fair amount of setup that is specific to BitmapHeapScan and once the state associated with that lives in the scan descriptor, it made more sense to make dedicated begin_scan()/rescan()/endscan() routines for BitmapHeapScan. Author: Melanie Plageman Discussion: https://postgr.es/m/CAAKRu_ZwCwWFeL_H3ia26bP2e7HiKLWt0ZmGXPVwPO6uXq0vaA%40mail.gmail.com --- src/backend/access/heap/heapam.c | 129 ++++++- src/backend/access/heap/heapam_handler.c | 306 ++++++++++++++-- src/backend/executor/nodeBitmapHeapscan.c | 410 +++------------------- src/include/access/heapam.h | 62 +++- src/include/access/relscan.h | 3 - src/include/access/tableam.h | 74 ++-- src/include/executor/nodeBitmapHeapscan.h | 6 + src/include/nodes/execnodes.h | 21 -- src/tools/pgindent/typedefs.list | 1 + 9 files changed, 553 insertions(+), 459 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 10f2faaa60..a21ec92d71 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -938,6 +938,48 @@ continue_page: * ---------------------------------------------------------------- */ +TableScanDesc +heap_beginscan_bm(Relation relation, Snapshot snapshot, uint32 flags) +{ + BitmapHeapScanDesc scan; + + /* + * increment relation ref count while scanning relation + * + * This is just to make really sure the relcache entry won't go away while + * the scan has a pointer to it. Caller should be holding the rel open + * anyway, so this is redundant in all normal scenarios... + */ + RelationIncrementReferenceCount(relation); + scan = (BitmapHeapScanDesc) palloc(sizeof(BitmapHeapScanDescData)); + + scan->heap_common.rs_base.rs_rd = relation; + scan->heap_common.rs_base.rs_snapshot = snapshot; + scan->heap_common.rs_base.rs_nkeys = 0; + scan->heap_common.rs_base.rs_flags = flags; + scan->heap_common.rs_base.rs_parallel = NULL; + scan->heap_common.rs_strategy = NULL; + + Assert(snapshot && IsMVCCSnapshot(snapshot)); + + /* we only need to set this up once */ + scan->heap_common.rs_ctup.t_tableOid = RelationGetRelid(relation); + + scan->heap_common.rs_parallelworkerdata = NULL; + scan->heap_common.rs_base.rs_key = NULL; + + initscan(&scan->heap_common, NULL, false); + + scan->iterator.serial = NULL; + scan->iterator.parallel = NULL; + scan->vmbuffer = InvalidBuffer; + scan->pf_iterator.serial = NULL; + scan->pf_iterator.parallel = NULL; + scan->pvmbuffer = InvalidBuffer; + + return (TableScanDesc) scan; +} + TableScanDesc heap_beginscan(Relation relation, Snapshot snapshot, @@ -967,8 +1009,6 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ - scan->rs_vmbuffer = InvalidBuffer; - scan->rs_empty_tuples_pending = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1026,6 +1066,35 @@ heap_beginscan(Relation relation, Snapshot snapshot, return (TableScanDesc) scan; } +/* + * Cleanup BitmapHeapScan table state + */ +void +heap_endscan_bm(TableScanDesc sscan) +{ + BitmapHeapScanDesc scan = (BitmapHeapScanDesc) sscan; + + if (BufferIsValid(scan->heap_common.rs_cbuf)) + ReleaseBuffer(scan->heap_common.rs_cbuf); + + if (BufferIsValid(scan->vmbuffer)) + ReleaseBuffer(scan->vmbuffer); + + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + + bhs_end_iterate(&scan->pf_iterator); + bhs_end_iterate(&scan->iterator); + + /* + * decrement relation reference count and free scan descriptor storage + */ + RelationDecrementReferenceCount(scan->heap_common.rs_base.rs_rd); + + pfree(scan); +} + + void heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, bool allow_strat, bool allow_sync, bool allow_pagemode) @@ -1057,12 +1126,6 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, if (BufferIsValid(scan->rs_cbuf)) ReleaseBuffer(scan->rs_cbuf); - if (BufferIsValid(scan->rs_vmbuffer)) - { - ReleaseBuffer(scan->rs_vmbuffer); - scan->rs_vmbuffer = InvalidBuffer; - } - /* * reinitialize scan descriptor */ @@ -1082,12 +1145,6 @@ heap_endscan(TableScanDesc sscan) if (BufferIsValid(scan->rs_cbuf)) ReleaseBuffer(scan->rs_cbuf); - if (BufferIsValid(scan->rs_vmbuffer)) - { - ReleaseBuffer(scan->rs_vmbuffer); - scan->rs_vmbuffer = InvalidBuffer; - } - /* * decrement relation reference count and free scan descriptor storage */ @@ -1334,6 +1391,50 @@ heap_getnextslot_tidrange(TableScanDesc sscan, ScanDirection direction, return true; } +void +heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, + ParallelBitmapHeapState *pstate, dsa_area *dsa, int pf_maximum) +{ + BitmapHeapScanDesc scan = (BitmapHeapScanDesc) sscan; + + if (BufferIsValid(scan->vmbuffer)) + ReleaseBuffer(scan->vmbuffer); + scan->vmbuffer = InvalidBuffer; + + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + + bhs_end_iterate(&scan->pf_iterator); + bhs_end_iterate(&scan->iterator); + + scan->prefetch_maximum = 0; + scan->empty_tuples_pending = 0; + scan->pstate = NULL; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + scan->pfblockno = InvalidBlockNumber; + + bhs_begin_iterate(&scan->iterator, + tbm, dsa, + pstate ? + pstate->tbmiterator : + InvalidDsaPointer); +#ifdef USE_PREFETCH + if (pf_maximum > 0) + { + bhs_begin_iterate(&scan->pf_iterator, tbm, dsa, + pstate ? + pstate->prefetch_iterator : + InvalidDsaPointer); + } +#endif /* USE_PREFETCH */ + + scan->pstate = pstate; + scan->prefetch_maximum = pf_maximum; +} + + /* * heap_fetch - retrieve tuple with given tid * diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 5e57d19d91..c513cbd2da 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -60,6 +60,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan); +static inline void BitmapPrefetch(BitmapHeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2186,12 +2189,80 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan) +{ +#ifdef USE_PREFETCH + BitmapHeapIterator *prefetch_iterator = &scan->pf_iterator; + ParallelBitmapHeapState *pstate = scan->pstate; + TBMIterateResult *tbmpre; + + if (pstate == NULL) + { + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (!prefetch_iterator->exhausted) + { + /* Do not let the prefetch iterator get behind the main one */ + tbmpre = bhs_iterate(prefetch_iterator); + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->prefetch_maximum > 0) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (!prefetch_iterator->exhausted) + { + tbmpre = bhs_iterate(prefetch_iterator); + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool -heapam_scan_bitmap_next_block(TableScanDesc scan, - BlockNumber *blockno, bool *recheck, +heapam_scan_bitmap_next_block(TableScanDesc sscan, + bool *recheck, long *lossy_pages, long *exact_pages) { - HeapScanDesc hscan = (HeapScanDesc) scan; + BitmapHeapScanDesc scan = (BitmapHeapScanDesc) sscan; + HeapScanDesc hscan = &scan->heap_common; BlockNumber block; Buffer buffer; Snapshot snapshot; @@ -2201,19 +2272,21 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, hscan->rs_cindex = 0; hscan->rs_ntuples = 0; - *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(scan); + do { CHECK_FOR_INTERRUPTS(); - tbmres = bhs_iterate(&scan->rs_bhs_iterator); + Assert(!scan->iterator.exhausted); + tbmres = bhs_iterate(&scan->iterator); if (tbmres == NULL) { /* no more entries in the bitmap */ - Assert(hscan->rs_empty_tuples_pending == 0); + Assert(scan->empty_tuples_pending == 0); return false; } @@ -2228,7 +2301,6 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, } while (!IsolationIsSerializable() && tbmres->blockno >= hscan->rs_nblocks); /* Got a valid block */ - *blockno = tbmres->blockno; *recheck = tbmres->recheck; /* @@ -2236,15 +2308,15 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, * heap, the bitmap entries don't need rechecking, and all tuples on the * page are visible to our transaction. */ - if (!(scan->rs_flags & SO_NEED_TUPLE) && + if (!(hscan->rs_base.rs_flags & SO_NEED_TUPLE) && !tbmres->recheck && - VM_ALL_VISIBLE(scan->rs_rd, tbmres->blockno, &hscan->rs_vmbuffer)) + VM_ALL_VISIBLE(hscan->rs_base.rs_rd, tbmres->blockno, &scan->vmbuffer)) { /* can't be lossy in the skip_fetch case */ Assert(tbmres->ntuples >= 0); - Assert(hscan->rs_empty_tuples_pending >= 0); + Assert(scan->empty_tuples_pending >= 0); - hscan->rs_empty_tuples_pending += tbmres->ntuples; + scan->empty_tuples_pending += tbmres->ntuples; return true; } @@ -2255,18 +2327,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, * Acquire pin on the target heap page, trading in any pin we held before. */ hscan->rs_cbuf = ReleaseAndReadBuffer(hscan->rs_cbuf, - scan->rs_rd, + hscan->rs_base.rs_rd, block); hscan->rs_cblock = block; buffer = hscan->rs_cbuf; - snapshot = scan->rs_snapshot; + snapshot = hscan->rs_base.rs_snapshot; ntup = 0; /* * Prune and repair fragmentation for the whole page, if possible. */ - heap_page_prune_opt(scan->rs_rd, buffer); + heap_page_prune_opt(hscan->rs_base.rs_rd, buffer); /* * We must hold share lock on the buffer content while examining tuple @@ -2294,7 +2366,7 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, HeapTupleData heapTuple; ItemPointerSet(&tid, block, offnum); - if (heap_hot_search_buffer(&tid, scan->rs_rd, buffer, snapshot, + if (heap_hot_search_buffer(&tid, hscan->rs_base.rs_rd, buffer, snapshot, &heapTuple, NULL, true)) hscan->rs_vistuples[ntup++] = ItemPointerGetOffsetNumber(&tid); } @@ -2320,16 +2392,16 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, continue; loctup.t_data = (HeapTupleHeader) PageGetItem(page, lp); loctup.t_len = ItemIdGetLength(lp); - loctup.t_tableOid = scan->rs_rd->rd_id; + loctup.t_tableOid = hscan->rs_base.rs_rd->rd_id; ItemPointerSet(&loctup.t_self, block, offnum); valid = HeapTupleSatisfiesVisibility(&loctup, snapshot, buffer); if (valid) { hscan->rs_vistuples[ntup++] = offnum; - PredicateLockTID(scan->rs_rd, &loctup.t_self, snapshot, + PredicateLockTID(hscan->rs_base.rs_rd, &loctup.t_self, snapshot, HeapTupleHeaderGetXmin(loctup.t_data)); } - HeapCheckForSerializableConflictOut(valid, scan->rs_rd, &loctup, + HeapCheckForSerializableConflictOut(valid, hscan->rs_base.rs_rd, &loctup, buffer, snapshot); } } @@ -2344,6 +2416,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->pstate == NULL && + !scan->pf_iterator.exhausted && + scan->pfblockno < block) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(scan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2354,22 +2438,168 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->pstate; + int prefetch_maximum = scan->prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(BitmapHeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->pstate; + BitmapHeapIterator *prefetch_iterator = &scan->pf_iterator; + Relation rel = scan->heap_common.rs_base.rs_rd; + + if (pstate == NULL) + { + if (!prefetch_iterator->exhausted) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator); + bool skip_fetch; + + if (tbmpre == NULL) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre->blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre->recheck && + VM_ALL_VISIBLE(rel, + tbmpre->blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + if (!prefetch_iterator->exhausted) + { + while (1) + { + TBMIterateResult *tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + tbmpre = bhs_iterate(prefetch_iterator); + if (tbmpre == NULL) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + break; + } + + scan->pfblockno = tbmpre->blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre->recheck && + VM_ALL_VISIBLE(rel, + tbmpre->blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) { - HeapScanDesc hscan = (HeapScanDesc) scan; + BitmapHeapScanDesc bscan = (BitmapHeapScanDesc) scan; + HeapScanDesc hscan = &bscan->heap_common; OffsetNumber targoffset; Page page; ItemId lp; - if (hscan->rs_empty_tuples_pending > 0) + if (bscan->empty_tuples_pending > 0) { /* * If we don't have to fetch the tuple, just return nulls. */ ExecStoreAllNullTuple(slot); - hscan->rs_empty_tuples_pending--; + bscan->empty_tuples_pending--; return true; } @@ -2379,6 +2609,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!bscan->pstate) + { + if (bscan->prefetch_target < bscan->prefetch_maximum) + bscan->prefetch_target++; + } + else if (bscan->pstate->prefetch_target < bscan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&bscan->pstate->mutex); + if (bscan->pstate->prefetch_target < bscan->prefetch_maximum) + bscan->pstate->prefetch_target++; + SpinLockRelease(&bscan->pstate->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(bscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); @@ -2704,6 +2964,10 @@ static const TableAmRoutine heapam_methods = { .scan_set_tidrange = heap_set_tidrange, .scan_getnextslot_tidrange = heap_getnextslot_tidrange, + .scan_rescan_bm = heap_rescan_bm, + .scan_begin_bm = heap_beginscan_bm, + .scan_end_bm = heap_endscan_bm, + .parallelscan_estimate = table_block_parallelscan_estimate, .parallelscan_initialize = table_block_parallelscan_initialize, .parallelscan_reinitialize = table_block_parallelscan_reinitialize, diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 70b560b8ee..d22a433f06 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); /* @@ -122,20 +118,9 @@ bhs_end_iterate(BitmapHeapIterator *iterator) static TupleTableSlot * BitmapHeapNext(BitmapHeapScanState *node) { - ExprContext *econtext; - TableScanDesc scan; - TIDBitmap *tbm; - TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; - dsa_area *dsa = node->ss.ps.state->es_query_dsa; - - /* - * extract necessary information from index scan node - */ - econtext = node->ss.ps.ps_ExprContext; - slot = node->ss.ss_ScanTupleSlot; - scan = node->ss.ss_currentScanDesc; - tbm = node->tbm; + ExprContext *econtext = node->ss.ps.ps_ExprContext; + TupleTableSlot *slot = node->ss.ss_ScanTupleSlot; + TableScanDesc scan = node->ss.ss_currentScanDesc; /* * If we haven't yet performed the underlying index scan, do it, and begin @@ -146,25 +131,37 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) { + Relation rel = node->ss.ss_currentRelation; + bool pf_maximum = 0; + bool init_shared_state = false; + uint32 extra_flags = 0; + /* * The leader will immediately come out of the function, but others * will be blocked until leader populates the TBM and wakes them up. */ - bool init_shared_state = node->pstate ? + init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; - if (!pstate || init_shared_state) + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(rel->rd_rel->reltablespace); +#endif + + if (!node->pstate || init_shared_state) { - tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); + node->tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); - if (!tbm || !IsA(tbm, TIDBitmap)) + if (!node->tbm || !IsA(node->tbm, TIDBitmap)) elog(ERROR, "unrecognized result from subplan"); - node->tbm = tbm; if (init_shared_state) { @@ -173,113 +170,52 @@ BitmapHeapNext(BitmapHeapScanState *node) * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(node->tbm); + #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = - tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; + node->pstate->prefetch_iterator = + tbm_prepare_shared_iterate(node->tbm); } #endif /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); + BitmapDoneInitializingSharedState(node->pstate); } } /* - * If this is the first scan of the underlying table, create the table - * scan descriptor and begin the scan. + * We can potentially skip fetching heap pages if we do not need any + * columns of the table, either for checking non-indexable quals or + * for returning data. This test is a bit simplistic, as it checks + * the stronger condition that there's no qual or return tlist at all. + * But in most cases it's probably not worth working harder than that. */ - if (!scan) - { - uint32 extra_flags = 0; - - /* - * We can potentially skip fetching heap pages if we do not need - * any columns of the table, either for checking non-indexable - * quals or for returning data. This test is a bit simplistic, as - * it checks the stronger condition that there's no qual or return - * tlist at all. But in most cases it's probably not worth working - * harder than that. - */ - if (node->ss.ps.plan->qual != NIL || node->ss.ps.plan->targetlist != NIL) - extra_flags |= SO_NEED_TUPLE; - - scan = table_beginscan_bm(node->ss.ss_currentRelation, - node->ss.ps.state->es_snapshot, - 0, - NULL, - extra_flags); - - node->ss.ss_currentScanDesc = scan; - } - - bhs_begin_iterate(&scan->rs_bhs_iterator, tbm, dsa, - pstate ? - pstate->tbmiterator : - InvalidDsaPointer); - -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) - { - bhs_begin_iterate(&node->pf_iterator, tbm, dsa, - pstate ? - pstate->prefetch_iterator : - InvalidDsaPointer); - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; - } -#endif /* USE_PREFETCH */ - + if (node->ss.ps.plan->qual != NIL || node->ss.ps.plan->targetlist != NIL) + extra_flags |= SO_NEED_TUPLE; + + scan = table_beginscan_bm(node->ss.ss_currentScanDesc, + rel, + node->ss.ps.state->es_snapshot, + extra_flags, + pf_maximum, + node->tbm, + node->pstate, + node->ss.ps.state->es_query_dsa); + + node->ss.ss_currentScanDesc = scan; node->initialized = true; goto new_page; } + for (;;) { while (table_scan_bitmap_next_tuple(scan, slot)) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -303,23 +239,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->blockno, &node->recheck, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - !node->pf_iterator.exhausted && - node->pfblockno < node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -343,215 +265,6 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = &node->pf_iterator; - TBMIterateResult *tbmpre; - - if (pstate == NULL) - { - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (!prefetch_iterator->exhausted) - { - /* Do not let the prefetch iterator get behind the main one */ - tbmpre = bhs_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (!prefetch_iterator->exhausted) - { - tbmpre = bhs_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = &node->pf_iterator; - - if (pstate == NULL) - { - if (!prefetch_iterator->exhausted) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator); - bool skip_fetch; - - if (tbmpre == NULL) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre->blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre->blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - if (!prefetch_iterator->exhausted) - { - while (1) - { - TBMIterateResult *tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - tbmpre = bhs_iterate(prefetch_iterator); - if (tbmpre == NULL) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - break; - } - - node->pfblockno = tbmpre->blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre->blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); - } - } - } -#endif /* USE_PREFETCH */ -} - /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual */ @@ -597,19 +310,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) if (node->ss.ss_currentScanDesc) table_rescan(node->ss.ss_currentScanDesc, NULL); - /* release bitmaps and buffers if any */ - if (node->pf_iterator.exhausted) - bhs_end_iterate(&node->pf_iterator); + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -648,14 +354,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (!node->pf_iterator.exhausted) - bhs_end_iterate(&node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -688,16 +390,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -737,13 +434,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -827,7 +517,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 750ea30852..73690d15c5 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -76,22 +76,60 @@ typedef struct HeapScanDescData */ ParallelBlockTableScanWorkerData *rs_parallelworkerdata; + /* these fields only used in page-at-a-time mode and for bitmap scans */ + int rs_cindex; /* current tuple's index in vistuples */ + int rs_ntuples; /* number of visible tuples on page */ + OffsetNumber rs_vistuples[MaxHeapTuplesPerPage]; /* their offsets */ + +} HeapScanDescData; +typedef struct HeapScanDescData *HeapScanDesc; + +typedef struct BitmapHeapScanDescData +{ + /* All the non-BitmapHeapScan specific members */ + HeapScanDescData heap_common; + + /* + * Members common to Parallel and Serial BitmapHeapScan + */ + struct BitmapHeapIterator iterator; + struct BitmapHeapIterator pf_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + /* * These fields are only used for bitmap scans for the "skip fetch" * optimization. Bitmap scans needing no fields from the heap may skip * fetching an all visible block, instead using the number of tuples per * block reported by the bitmap to determine how many NULL-filled tuples - * to return. + * to return. They are common to parallel and serial BitmapHeapScans */ - Buffer rs_vmbuffer; - int rs_empty_tuples_pending; + Buffer vmbuffer; + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + int empty_tuples_pending; + + /* + * Parallel-only members + */ + + struct ParallelBitmapHeapState *pstate; + + /* + * Serial-only members + */ + + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; +} BitmapHeapScanDescData; + +typedef struct BitmapHeapScanDescData *BitmapHeapScanDesc; - /* these fields only used in page-at-a-time mode and for bitmap scans */ - int rs_cindex; /* current tuple's index in vistuples */ - int rs_ntuples; /* number of visible tuples on page */ - OffsetNumber rs_vistuples[MaxHeapTuplesPerPage]; /* their offsets */ -} HeapScanDescData; -typedef struct HeapScanDescData *HeapScanDesc; /* * Descriptor for fetches from heap via an index. @@ -289,6 +327,12 @@ extern void heap_set_tidrange(TableScanDesc sscan, ItemPointer mintid, extern bool heap_getnextslot_tidrange(TableScanDesc sscan, ScanDirection direction, TupleTableSlot *slot); +extern TableScanDesc heap_beginscan_bm(Relation relation, Snapshot snapshot, uint32 flags); + +extern void heap_endscan_bm(TableScanDesc scan); +extern void heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, + ParallelBitmapHeapState *pstate, dsa_area *dsa, int pf_maximum); + extern bool heap_fetch(Relation relation, Snapshot snapshot, HeapTuple tuple, Buffer *userbuf, bool keep_buf); extern bool heap_hot_search_buffer(ItemPointer tid, Relation relation, diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index e520186b41..855b9558bf 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -41,9 +41,6 @@ typedef struct TableScanDescData ItemPointerData rs_mintid; ItemPointerData rs_maxtid; - /* Only used for Bitmap table scans */ - BitmapHeapIterator rs_bhs_iterator; - /* * Information about type and behaviour of the scan, a bitmask of members * of the ScanOptions enum (see tableam.h). diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 618ab2b449..138bf5f6ed 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -391,6 +391,23 @@ typedef struct TableAmRoutine ScanDirection direction, TupleTableSlot *slot); + /* + * TODO: add comment + */ + TableScanDesc (*scan_begin_bm) (Relation rel, + Snapshot snapshot, + uint32 flags); + + void (*scan_rescan_bm) (TableScanDesc scan, TIDBitmap *tbm, + ParallelBitmapHeapState *pstate, dsa_area *dsa, + int pf_maximum); + + /* + * Release resources and deallocate scan. If TableScanDesc.temp_snap, + * TableScanDesc.rs_snapshot needs to be unregistered. + */ + void (*scan_end_bm) (TableScanDesc scan); + /* ------------------------------------------------------------------------ * Parallel table scan related functions. * ------------------------------------------------------------------------ @@ -774,9 +791,9 @@ typedef struct TableAmRoutine */ /* - * Prepare to fetch / check / return tuples from `blockno` as part of a - * bitmap table scan. `scan` was started via table_beginscan_bm(). Return - * false if the bitmap is exhausted and true otherwise. + * Prepare to fetch / check / return tuples as part of a bitmap table + * scan. `scan` was started via table_beginscan_bm(). Return false if the + * bitmap is exhausted and true otherwise. * * This will typically read and pin the target block, and do the necessary * work to allow scan_bitmap_next_tuple() to return tuples (e.g. it might @@ -785,22 +802,11 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the bitmap is lossy for the selected * block; otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ bool (*scan_bitmap_next_block) (TableScanDesc scan, - BlockNumber *blockno, bool *recheck, + bool *recheck, long *lossy_pages, long *exact_pages); /* @@ -929,18 +935,26 @@ table_beginscan_strat(Relation rel, Snapshot snapshot, } /* - * table_beginscan_bm is an alternative entry point for setting up a - * TableScanDesc for a bitmap heap scan. Although that scan technology is - * really quite unlike a standard seqscan, there is just enough commonality to - * make it worth using the same data structure. + * table_beginscan_bm is the entry point for setting up a TableScanDesc for a + * bitmap heap scan. */ static inline TableScanDesc -table_beginscan_bm(Relation rel, Snapshot snapshot, - int nkeys, struct ScanKeyData *key, uint32 extra_flags) +table_beginscan_bm(TableScanDesc scan, Relation rel, Snapshot snapshot, + uint32 extra_flags, int pf_maximum, TIDBitmap *tbm, + ParallelBitmapHeapState *pstate, dsa_area *dsa) { uint32 flags = SO_TYPE_BITMAPSCAN | SO_ALLOW_PAGEMODE | extra_flags; - return rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); + /* + * If this is the first scan of the underlying table, create the table + * scan descriptor and begin the scan. + */ + if (!scan) + scan = rel->rd_tableam->scan_begin_bm(rel, snapshot, flags); + + scan->rs_rd->rd_tableam->scan_rescan_bm(scan, tbm, pstate, dsa, pf_maximum); + + return scan; } /* @@ -987,9 +1001,11 @@ table_beginscan_tid(Relation rel, Snapshot snapshot) static inline void table_endscan(TableScanDesc scan) { - if (scan->rs_flags & SO_TYPE_BITMAPSCAN && - !scan->rs_bhs_iterator.exhausted) - bhs_end_iterate(&scan->rs_bhs_iterator); + if (scan->rs_flags & SO_TYPE_BITMAPSCAN) + { + scan->rs_rd->rd_tableam->scan_end_bm(scan); + return; + } scan->rs_rd->rd_tableam->scan_end(scan); } @@ -1001,10 +1017,6 @@ static inline void table_rescan(TableScanDesc scan, struct ScanKeyData *key) { - if (scan->rs_flags & SO_TYPE_BITMAPSCAN && - !scan->rs_bhs_iterator.exhausted) - bhs_end_iterate(&scan->rs_bhs_iterator); - scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); } @@ -1979,7 +1991,7 @@ table_relation_estimate_size(Relation rel, int32 *attr_widths, */ static inline bool table_scan_bitmap_next_block(TableScanDesc scan, - BlockNumber *blockno, bool *recheck, + bool *recheck, long *lossy_pages, long *exact_pages) { @@ -1992,7 +2004,7 @@ table_scan_bitmap_next_block(TableScanDesc scan, elog(ERROR, "unexpected table_scan_bitmap_next_block call during logical decoding"); return scan->rs_rd->rd_tableam->scan_bitmap_next_block(scan, - blockno, recheck, + recheck, lossy_pages, exact_pages); } diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h index 7064f54686..f3c92f7a8a 100644 --- a/src/include/executor/nodeBitmapHeapscan.h +++ b/src/include/executor/nodeBitmapHeapscan.h @@ -28,6 +28,12 @@ extern void ExecBitmapHeapReInitializeDSM(BitmapHeapScanState *node, ParallelContext *pcxt); extern void ExecBitmapHeapInitializeWorker(BitmapHeapScanState *node, ParallelWorkerContext *pwcxt); +typedef struct BitmapHeapIterator +{ + struct TBMIterator *serial; + struct TBMSharedIterator *parallel; + bool exhausted; +} BitmapHeapIterator; extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator); extern void bhs_begin_iterate(BitmapHeapIterator *iterator, TIDBitmap *tbm, diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 714eb3e534..49ce4ff9a7 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1787,30 +1787,16 @@ typedef struct ParallelBitmapHeapState ConditionVariable cv; } ParallelBitmapHeapState; -typedef struct BitmapHeapIterator -{ - struct TBMIterator *serial; - struct TBMSharedIterator *parallel; - bool exhausted; -} BitmapHeapIterator; - /* ---------------- * BitmapHeapScanState information * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1818,18 +1804,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - BitmapHeapIterator pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 2348a8793e..443b01c053 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -263,6 +263,7 @@ BitmapHeapIterator BitmapHeapPath BitmapHeapScan BitmapHeapScanState +BitmapHeapScanDesc BitmapIndexScan BitmapIndexScanState BitmapOr -- 2.40.1 --pbix6fw3h4kvmjae Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v15-0012-Move-BitmapHeapScan-initialization-to-helper.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v10 14/17] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 262 +++++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 341 ++-------------------- src/include/access/heapam.h | 17 ++ src/include/access/relscan.h | 8 + src/include/access/tableam.h | 26 +- src/include/nodes/execnodes.h | 14 - 7 files changed, 355 insertions(+), 339 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index 6b2863391f..3d92fb5135 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -948,8 +948,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1032,6 +1040,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1044,6 +1058,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1069,6 +1089,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 105137396b..867061325c 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -55,6 +55,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2112,6 +2115,73 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + bhs_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2130,6 +2200,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2270,6 +2342,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->rs_pf_bhs_iterator && + hscan->pfblockno > hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2280,6 +2364,154 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + BitmapHeapIterator *prefetch_iterator = scan->rs_base.rs_pf_bhs_iterator; + + if (pstate == NULL) + { + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + bhs_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + bhs_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + bhs_end_iterate(prefetch_iterator); + scan->rs_base.rs_pf_bhs_iterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2305,6 +2537,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 78f79aafff..187b288e68 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); static BitmapHeapIterator *bhs_begin_iterate(TIDBitmap *tbm, dsa_pointer shared_area, @@ -122,7 +118,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -142,7 +137,7 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) @@ -154,7 +149,16 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; - if (!pstate || init_shared_state) + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif + + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); @@ -169,23 +173,16 @@ BitmapHeapNext(BitmapHeapScanState *node) * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); + BitmapDoneInitializingSharedState(node->pstate); } } @@ -216,19 +213,19 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + scan->rs_bhs_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->tbmiterator : InvalidDsaPointer, + scan->bm_parallel ? scan->bm_parallel->tbmiterator : InvalidDsaPointer, dsa); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { - node->pf_iterator = bhs_begin_iterate(tbm, - pstate ? pstate->prefetch_iterator : InvalidDsaPointer, - dsa); - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; + scan->rs_pf_bhs_iterator = bhs_begin_iterate(tbm, + scan->bm_parallel ? scan->bm_parallel->prefetch_iterator : InvalidDsaPointer, + dsa); } #endif /* USE_PREFETCH */ @@ -243,36 +240,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -296,23 +263,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->pf_iterator && - node->pfblockno > node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -336,219 +289,6 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - bhs_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = node->pf_iterator; - - if (pstate == NULL) - { - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - bhs_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - bhs_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - bhs_end_iterate(prefetch_iterator); - node->pf_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} - /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual */ @@ -594,22 +334,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) if (node->ss.ss_currentScanDesc) table_rescan(node->ss.ss_currentScanDesc, NULL); - /* release bitmaps and buffers if any */ - if (node->pf_iterator) - { - bhs_end_iterate(node->pf_iterator); - node->pf_iterator = NULL; - } + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -648,14 +378,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (node->pf_iterator) - bhs_end_iterate(node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -688,17 +414,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->pf_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -738,13 +458,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -828,7 +541,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index cef54e2d5d..29fdd55893 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -82,6 +82,23 @@ typedef struct HeapScanDescData Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* + * These fields only used for prefetching in bitmap table scans + */ + + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used in serial BHS + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; + /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ int rs_ntuples; /* number of visible tuples on page */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index fb22f305bf..7938b741d6 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct BitmapHeapIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -45,6 +46,13 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct BitmapHeapIterator *rs_bhs_iterator; + struct BitmapHeapIterator *rs_pf_bhs_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + /* used to validate BHS prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index edc54ecffe..284ea3d864 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -800,17 +800,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -960,6 +949,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->rs_bhs_iterator = NULL; + result->rs_pf_bhs_iterator = NULL; + result->prefetch_maximum = 0; + result->bm_parallel = NULL; return result; } @@ -1024,6 +1016,12 @@ table_endscan(TableScanDesc scan) { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1040,6 +1038,12 @@ table_rescan(TableScanDesc scan, { bhs_end_iterate(scan->rs_bhs_iterator); scan->rs_bhs_iterator = NULL; + + if (scan->rs_pf_bhs_iterator) + { + bhs_end_iterate(scan->rs_pf_bhs_iterator); + scan->rs_pf_bhs_iterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 52cedd1b35..60916bf0d0 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1785,18 +1785,11 @@ struct BitmapHeapIterator; * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1804,18 +1797,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - struct BitmapHeapIterator *pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --3o7pc6dfau5a5hry Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0015-Remove-table_scan_bitmap_next_block.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v9 13/17] Push BitmapHeapScan prefetch code into heapam.c @ 2024-03-22 13:42 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-03-22 13:42 UTC (permalink / raw) In preparation for transitioning to using the streaming read API for prefetching [1], move all of the BitmapHeapScanState members related to prefetching and the functions for accessing them into the HeapScanDescData and TableScanDescData. Members that still need to be accessed in BitmapHeapNext() could not be moved into heap AM-specific code. Specifically, parallel iterator setup requires several components which seem odd to pass to the table AM API. [1] https://www.postgresql.org/message-id/flat/CA%2BhUKGJkOiOCa%2Bmag4BF%2BzHo7qo%3Do9CFheB8%3Dg6uT5TUm2... --- src/backend/access/heap/heapam.c | 26 ++ src/backend/access/heap/heapam_handler.c | 268 +++++++++++++++ src/backend/executor/nodeBitmapHeapscan.c | 397 +++------------------- src/include/access/heapam.h | 12 + src/include/access/relscan.h | 11 + src/include/access/tableam.h | 38 ++- src/include/nodes/execnodes.h | 16 - 7 files changed, 388 insertions(+), 380 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index e7bed84f75..c12563a188 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -951,8 +951,16 @@ heap_beginscan(Relation relation, Snapshot snapshot, scan->rs_base.rs_flags = flags; scan->rs_base.rs_parallel = parallel_scan; scan->rs_strategy = NULL; /* set in initscan */ + + scan->rs_base.blockno = InvalidBlockNumber; + scan->rs_vmbuffer = InvalidBuffer; scan->rs_empty_tuples_pending = 0; + scan->pvmbuffer = InvalidBuffer; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; /* * Disable page-at-a-time mode if it's not a MVCC-safe snapshot. @@ -1035,6 +1043,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_base.rs_flags &= ~SO_ALLOW_PAGEMODE; } + scan->rs_base.blockno = InvalidBlockNumber; + + scan->pfblockno = InvalidBlockNumber; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + /* * unpin scan buffers */ @@ -1047,6 +1061,12 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * reinitialize scan descriptor */ @@ -1072,6 +1092,12 @@ heap_endscan(TableScanDesc sscan) scan->rs_vmbuffer = InvalidBuffer; } + if (BufferIsValid(scan->pvmbuffer)) + { + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + } + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index adfc77684a..efd2784e03 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -55,6 +55,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(HeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(HeapScanDesc scan); +static inline void BitmapPrefetch(HeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2112,6 +2115,76 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * ------------------------------------------------------------------------ */ +/* + * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator + * + * We keep track of how far the prefetch iterator is ahead of the main + * iterator in prefetch_pages. For each block the main iterator returns, we + * decrement prefetch_pages. + */ +static inline void +BitmapAdjustPrefetchIterator(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + TBMIterateResult tbmpre; + + if (pstate == NULL) + { + TBMIterator *prefetch_iterator = scan->rs_base.pf_tbmiterator; + + if (scan->prefetch_pages > 0) + { + /* The main iterator has closed the distance by one page */ + scan->prefetch_pages--; + } + else if (prefetch_iterator) + { + /* Do not let the prefetch iterator get behind the main one */ + tbm_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + return; + } + + /* + * Adjusting the prefetch iterator before invoking + * table_scan_bitmap_next_block() keeps prefetch distance higher across + * the parallel workers. + */ + if (scan->rs_base.prefetch_maximum > 0) + { + TBMSharedIterator *prefetch_iterator = scan->rs_base.pf_shared_tbmiterator; + + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages > 0) + { + pstate->prefetch_pages--; + SpinLockRelease(&pstate->mutex); + } + else + { + /* Release the mutex before iterating */ + SpinLockRelease(&pstate->mutex); + + /* + * In case of shared mode, we can not ensure that the current + * blockno of the main iterator and that of the prefetch iterator + * are same. It's possible that whatever blockno we are + * prefetching will be processed by another process. Therefore, + * we don't validate the blockno here as we do in non-parallel + * case. + */ + if (prefetch_iterator) + { + tbm_shared_iterate(prefetch_iterator, &tbmpre); + scan->pfblockno = tbmpre.blockno; + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_block(TableScanDesc scan, bool *recheck, BlockNumber *blockno, @@ -2130,6 +2203,8 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(hscan); + do { CHECK_FOR_INTERRUPTS(); @@ -2273,6 +2348,18 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->bm_parallel == NULL && + scan->pf_tbmiterator && + hscan->pfblockno > hscan->rs_base.blockno) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(hscan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2283,6 +2370,157 @@ heapam_scan_bitmap_next_block(TableScanDesc scan, return true; } +/* + * BitmapAdjustPrefetchTarget - Adjust the prefetch target + * + * Increase prefetch target if it's not yet at the max. Note that + * we will increase it to zero after fetching the very first + * page/tuple, then to one after the second tuple is fetched, then + * it doubles as later pages are fetched. + */ +static inline void +BitmapAdjustPrefetchTarget(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + int prefetch_maximum = scan->rs_base.prefetch_maximum; + + if (pstate == NULL) + { + if (scan->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; + else + scan->prefetch_target++; + return; + } + + /* Do an unlocked check first to save spinlock acquisitions. */ + if (pstate->prefetch_target < prefetch_maximum) + { + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_target >= prefetch_maximum) + /* don't increase any further */ ; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; + else if (pstate->prefetch_target > 0) + pstate->prefetch_target *= 2; + else + pstate->prefetch_target++; + SpinLockRelease(&pstate->mutex); + } +#endif /* USE_PREFETCH */ +} + + +/* + * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target + */ +static inline void +BitmapPrefetch(HeapScanDesc scan) +{ +#ifdef USE_PREFETCH + ParallelBitmapHeapState *pstate = scan->rs_base.bm_parallel; + + if (pstate == NULL) + { + TBMIterator *prefetch_iterator = scan->rs_base.pf_tbmiterator; + + if (prefetch_iterator) + { + while (scan->prefetch_pages < scan->prefetch_target) + { + TBMIterateResult tbmpre; + bool skip_fetch; + + tbm_iterate(prefetch_iterator, &tbmpre); + + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + tbm_end_iterate(prefetch_iterator); + scan->rs_base.pf_tbmiterator = NULL; + break; + } + scan->prefetch_pages++; + scan->pfblockno = tbmpre.blockno; + + /* + * If we expect not to have to actually read this heap page, + * skip this prefetch call, but continue to run the prefetch + * logic normally. (Would it be better not to increment + * prefetch_pages?) + */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + + return; + } + + if (pstate->prefetch_pages < pstate->prefetch_target) + { + TBMSharedIterator *prefetch_iterator = scan->rs_base.pf_shared_tbmiterator; + + if (prefetch_iterator) + { + while (1) + { + TBMIterateResult tbmpre; + bool do_prefetch = false; + bool skip_fetch; + + /* + * Recheck under the mutex. If some other process has already + * done enough prefetching then we need not to do anything. + */ + SpinLockAcquire(&pstate->mutex); + if (pstate->prefetch_pages < pstate->prefetch_target) + { + pstate->prefetch_pages++; + do_prefetch = true; + } + SpinLockRelease(&pstate->mutex); + + if (!do_prefetch) + return; + + tbm_shared_iterate(prefetch_iterator, &tbmpre); + if (!BlockNumberIsValid(tbmpre.blockno)) + { + /* No more pages to prefetch */ + tbm_end_shared_iterate(prefetch_iterator); + scan->rs_base.pf_shared_tbmiterator = NULL; + break; + } + + scan->pfblockno = tbmpre.blockno; + + /* As above, skip prefetch if we expect not to need page */ + skip_fetch = (!(scan->rs_base.rs_flags & SO_NEED_TUPLE) && + !tbmpre.recheck && + VM_ALL_VISIBLE(scan->rs_base.rs_rd, + tbmpre.blockno, + &scan->pvmbuffer)); + + if (!skip_fetch) + PrefetchBuffer(scan->rs_base.rs_rd, MAIN_FORKNUM, tbmpre.blockno); + } + } + } +#endif /* USE_PREFETCH */ +} + static bool heapam_scan_bitmap_next_tuple(TableScanDesc scan, TupleTableSlot *slot) @@ -2308,6 +2546,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->bm_parallel) + { + if (hscan->prefetch_target < scan->prefetch_maximum) + hscan->prefetch_target++; + } + else if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->bm_parallel->mutex); + if (scan->bm_parallel->prefetch_target < scan->prefetch_maximum) + scan->bm_parallel->prefetch_target++; + SpinLockRelease(&scan->bm_parallel->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(hscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 51c4360205..f241f4cb2c 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -51,10 +51,6 @@ static TupleTableSlot *BitmapHeapNext(BitmapHeapScanState *node); static inline void BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate); -static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -static inline void BitmapPrefetch(BitmapHeapScanState *node, - TableScanDesc scan); static bool BitmapShouldInitializeSharedState(ParallelBitmapHeapState *pstate); @@ -71,7 +67,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TableScanDesc scan; TIDBitmap *tbm; TupleTableSlot *slot; - ParallelBitmapHeapState *pstate = node->pstate; dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* @@ -91,83 +86,53 @@ BitmapHeapNext(BitmapHeapScanState *node) * prefetching. node->prefetch_pages tracks exactly how many pages ahead * the prefetch iterator is. Also, node->prefetch_target tracks the * desired prefetch distance, which starts small and increases up to the - * node->prefetch_maximum. This is to avoid doing a lot of prefetching in + * scan->prefetch_maximum. This is to avoid doing a lot of prefetching in * a scan that stops after a few tuples because of a LIMIT. */ if (!node->initialized) { - TBMIterator *tbmiterator = NULL; - TBMSharedIterator *shared_tbmiterator = NULL; + /* + * The leader will immediately come out of the function, but others + * will be blocked until leader populates the TBM and wakes them up. + */ + bool init_shared_state = node->pstate ? + BitmapShouldInitializeSharedState(node->pstate) : false; + + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ + int pf_maximum = 0; +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(node->ss.ss_currentRelation->rd_rel->reltablespace); +#endif - if (!pstate) + if (!node->pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); if (!tbm || !IsA(tbm, TIDBitmap)) elog(ERROR, "unrecognized result from subplan"); - node->tbm = tbm; - tbmiterator = tbm_begin_iterate(tbm); -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (init_shared_state) { - node->prefetch_iterator = tbm_begin_iterate(tbm); - node->prefetch_pages = 0; - node->prefetch_target = -1; - } -#endif /* USE_PREFETCH */ - } - else - { - /* - * The leader will immediately come out of the function, but - * others will be blocked until leader populates the TBM and wakes - * them up. - */ - if (BitmapShouldInitializeSharedState(pstate)) - { - tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); - if (!tbm || !IsA(tbm, TIDBitmap)) - elog(ERROR, "unrecognized result from subplan"); - - node->tbm = tbm; - /* * Prepare to iterate over the TBM. This will return the * dsa_pointer of the iterator state which will be used by * multiple processes to iterate jointly. */ - pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); + node->pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { - pstate->prefetch_iterator = + node->pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); - - /* - * We don't need the mutex here as we haven't yet woke up - * others. - */ - pstate->prefetch_pages = 0; - pstate->prefetch_target = -1; } #endif - /* We have initialized the shared state so wake up others. */ - BitmapDoneInitializingSharedState(pstate); - } - - /* Allocate a private iterator and attach the shared state to it */ - shared_tbmiterator = tbm_attach_shared_iterate(dsa, pstate->tbmiterator); - -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) - { - node->shared_prefetch_iterator = - tbm_attach_shared_iterate(dsa, pstate->prefetch_iterator); + BitmapDoneInitializingSharedState(node->pstate); } -#endif /* USE_PREFETCH */ } /* @@ -197,8 +162,26 @@ BitmapHeapNext(BitmapHeapScanState *node) extra_flags); } - scan->tbmiterator = tbmiterator; - scan->shared_tbmiterator = shared_tbmiterator; + scan->prefetch_maximum = pf_maximum; + scan->bm_parallel = node->pstate; + + if (!scan->bm_parallel) + scan->tbmiterator = tbm_begin_iterate(tbm); + else + /* Allocate a private iterator and attach the shared state to it */ + scan->shared_tbmiterator = tbm_attach_shared_iterate(dsa, scan->bm_parallel->tbmiterator); + +#ifdef USE_PREFETCH + if (scan->prefetch_maximum > 0) + { + if (!scan->bm_parallel) + scan->pf_tbmiterator = tbm_begin_iterate(tbm); + else + scan->pf_shared_tbmiterator = + tbm_attach_shared_iterate(dsa, scan->bm_parallel->prefetch_iterator); + } +#endif /* USE_PREFETCH */ + node->initialized = true; @@ -211,36 +194,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); /* * If we are using lossy info, we have to recheck the qual @@ -264,23 +217,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - - if (!table_scan_bitmap_next_block(scan, &node->recheck, &node->blockno, + if (!table_scan_bitmap_next_block(scan, &node->recheck, &scan->blockno, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - node->prefetch_iterator && - node->pfblockno > node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -304,224 +243,7 @@ BitmapDoneInitializingSharedState(ParallelBitmapHeapState *pstate) ConditionVariableBroadcast(&pstate->cv); } -/* - * BitmapAdjustPrefetchIterator - Adjust the prefetch iterator - * - * We keep track of how far the prefetch iterator is ahead of the main - * iterator in prefetch_pages. For each block the main iterator returns, we - * decrement prefetch_pages. - */ -static inline void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - TBMIterateResult tbmpre; - - if (pstate == NULL) - { - TBMIterator *prefetch_iterator = node->prefetch_iterator; - - if (node->prefetch_pages > 0) - { - /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; - } - else if (prefetch_iterator) - { - /* Do not let the prefetch iterator get behind the main one */ - tbm_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - return; - } - - /* - * Adjusting the prefetch iterator before invoking - * table_scan_bitmap_next_block() keeps prefetch distance higher across - * the parallel workers. - */ - if (node->prefetch_maximum > 0) - { - TBMSharedIterator *prefetch_iterator = node->shared_prefetch_iterator; - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages > 0) - { - pstate->prefetch_pages--; - SpinLockRelease(&pstate->mutex); - } - else - { - /* Release the mutex before iterating */ - SpinLockRelease(&pstate->mutex); - - /* - * In case of shared mode, we can not ensure that the current - * blockno of the main iterator and that of the prefetch iterator - * are same. It's possible that whatever blockno we are - * prefetching will be processed by another process. Therefore, - * we don't validate the blockno here as we do in non-parallel - * case. - */ - if (prefetch_iterator) - { - tbm_shared_iterate(prefetch_iterator, &tbmpre); - node->pfblockno = tbmpre.blockno; - } - } - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapAdjustPrefetchTarget - Adjust the prefetch target - * - * Increase prefetch target if it's not yet at the max. Note that - * we will increase it to zero after fetching the very first - * page/tuple, then to one after the second tuple is fetched, then - * it doubles as later pages are fetched. - */ -static inline void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - if (node->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; - else - node->prefetch_target++; - return; - } - - /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) - { - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) - /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; - else if (pstate->prefetch_target > 0) - pstate->prefetch_target *= 2; - else - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ -} - -/* - * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target - */ -static inline void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) -{ -#ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - - if (pstate == NULL) - { - TBMIterator *prefetch_iterator = node->prefetch_iterator; - - if (prefetch_iterator) - { - while (node->prefetch_pages < node->prefetch_target) - { - TBMIterateResult tbmpre; - bool skip_fetch; - - tbm_iterate(prefetch_iterator, &tbmpre); - - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - tbm_end_iterate(prefetch_iterator); - node->prefetch_iterator = NULL; - break; - } - node->prefetch_pages++; - node->pfblockno = tbmpre.blockno; - - /* - * If we expect not to have to actually read this heap page, - * skip this prefetch call, but continue to run the prefetch - * logic normally. (Would it be better not to increment - * prefetch_pages?) - */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - - return; - } - - if (pstate->prefetch_pages < pstate->prefetch_target) - { - TBMSharedIterator *prefetch_iterator = node->shared_prefetch_iterator; - - if (prefetch_iterator) - { - while (1) - { - TBMIterateResult tbmpre; - bool do_prefetch = false; - bool skip_fetch; - - /* - * Recheck under the mutex. If some other process has already - * done enough prefetching then we need not to do anything. - */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_pages < pstate->prefetch_target) - { - pstate->prefetch_pages++; - do_prefetch = true; - } - SpinLockRelease(&pstate->mutex); - - if (!do_prefetch) - return; - - tbm_shared_iterate(prefetch_iterator, &tbmpre); - if (!BlockNumberIsValid(tbmpre.blockno)) - { - /* No more pages to prefetch */ - tbm_end_shared_iterate(prefetch_iterator); - node->shared_prefetch_iterator = NULL; - break; - } - - node->pfblockno = tbmpre.blockno; - - /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && - !tbmpre.recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, - tbmpre.blockno, - &node->pvmbuffer)); - - if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre.blockno); - } - } - } -#endif /* USE_PREFETCH */ -} /* * BitmapHeapRecheck -- access method routine to recheck a tuple in EvalPlanQual @@ -569,22 +291,11 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) table_rescan(node->ss.ss_currentScanDesc, NULL); /* release bitmaps and buffers if any */ - if (node->prefetch_iterator) - tbm_end_iterate(node->prefetch_iterator); - if (node->shared_prefetch_iterator) - tbm_end_shared_iterate(node->shared_prefetch_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; - node->prefetch_iterator = NULL; node->initialized = false; - node->shared_prefetch_iterator = NULL; - node->pvmbuffer = InvalidBuffer; node->recheck = true; - node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; ExecScanReScan(&node->ss); @@ -625,14 +336,8 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) /* * release bitmaps and buffers if any */ - if (node->prefetch_iterator) - tbm_end_iterate(node->prefetch_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->shared_prefetch_iterator) - tbm_end_shared_iterate(node->shared_prefetch_iterator); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -665,18 +370,11 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->prefetch_iterator = NULL; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = 0; scanstate->initialized = false; - scanstate->shared_prefetch_iterator = NULL; scanstate->pstate = NULL; scanstate->recheck = true; - scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -716,13 +414,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* @@ -806,7 +497,7 @@ ExecBitmapHeapInitializeDSM(BitmapHeapScanState *node, /* Initialize the mutex */ SpinLockInit(&pstate->mutex); pstate->prefetch_pages = 0; - pstate->prefetch_target = 0; + pstate->prefetch_target = -1; pstate->state = BM_INITIAL; ConditionVariableInit(&pstate->cv); diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 3dfb19ec7d..22bdccc2a9 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -81,6 +81,18 @@ typedef struct HeapScanDescData */ Buffer rs_vmbuffer; int rs_empty_tuples_pending; + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; + + /* + * These fields only used for prefetching in bitmap table scans + */ + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; /* these fields only used in page-at-a-time mode and for bitmap scans */ int rs_cindex; /* current tuple's index in vistuples */ diff --git a/src/include/access/relscan.h b/src/include/access/relscan.h index 92b829cebc..93168bd350 100644 --- a/src/include/access/relscan.h +++ b/src/include/access/relscan.h @@ -26,6 +26,7 @@ struct ParallelTableScanDescData; struct TBMIterator; struct TBMSharedIterator; +struct ParallelBitmapHeapState; /* * Generic descriptor for table scans. This is the base-class for table scans, @@ -46,6 +47,16 @@ typedef struct TableScanDescData /* Only used for Bitmap table scans */ struct TBMIterator *tbmiterator; struct TBMSharedIterator *shared_tbmiterator; + /* Prefetch iterators */ + struct TBMIterator *pf_tbmiterator; + struct TBMSharedIterator *pf_shared_tbmiterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; + struct ParallelBitmapHeapState *bm_parallel; + + /* used to validate prefetch and current block stay in sync */ + BlockNumber blockno; /* * Information about type and behaviour of the scan, a bitmask of members diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 1d4b79a73f..9cab4462d6 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -800,17 +800,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the block's representation in the bitmap * is lossy, otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -961,6 +950,9 @@ table_beginscan_bm(Relation rel, Snapshot snapshot, result = rel->rd_tableam->scan_begin(rel, snapshot, nkeys, key, NULL, flags); result->shared_tbmiterator = NULL; result->tbmiterator = NULL; + result->pf_shared_tbmiterator = NULL; + result->pf_tbmiterator = NULL; + result->bm_parallel = NULL; return result; } @@ -1029,11 +1021,23 @@ table_endscan(TableScanDesc scan) scan->shared_tbmiterator = NULL; } + if (scan->pf_shared_tbmiterator) + { + tbm_end_shared_iterate(scan->pf_shared_tbmiterator); + scan->pf_shared_tbmiterator = NULL; + } + if (scan->tbmiterator) { tbm_end_iterate(scan->tbmiterator); scan->tbmiterator = NULL; } + + if (scan->pf_tbmiterator) + { + tbm_end_iterate(scan->pf_tbmiterator); + scan->pf_tbmiterator = NULL; + } } scan->rs_rd->rd_tableam->scan_end(scan); @@ -1054,11 +1058,23 @@ table_rescan(TableScanDesc scan, scan->shared_tbmiterator = NULL; } + if (scan->pf_shared_tbmiterator) + { + tbm_end_shared_iterate(scan->pf_shared_tbmiterator); + scan->pf_shared_tbmiterator = NULL; + } + if (scan->tbmiterator) { tbm_end_iterate(scan->tbmiterator); scan->tbmiterator = NULL; } + + if (scan->pf_tbmiterator) + { + tbm_end_iterate(scan->pf_tbmiterator); + scan->pf_tbmiterator = NULL; + } } scan->rs_rd->rd_tableam->scan_rescan(scan, key, false, false, false, false); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 8688bc5ab0..7a3fdf9cd4 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1783,19 +1783,11 @@ typedef struct ParallelBitmapHeapState * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * prefetch_iterator iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate - * shared_prefetch_iterator shared iterator for prefetching * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck - * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1803,19 +1795,11 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - TBMIterator *prefetch_iterator; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - TBMSharedIterator *shared_prefetch_iterator; ParallelBitmapHeapState *pstate; bool recheck; - BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --7mdtsjmrzitrgzgx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0014-Unify-parallel-and-serial-BitmapHeapScan-iterator.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v16 16/18] Push BitmapHeapScan prefetch code into heap AM @ 2024-04-05 22:48 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-04-05 22:48 UTC (permalink / raw) An earlier commit eliminated a table AM layering violation for the current block in a BitmapHeapScan but left the violation in BitmapHeapScan prefetching code. To resolve this, move and manage all state used for prefetching entirely into the heap AM. Author: Melanie Plageman Discussion: https://postgr.es/m/CAAKRu_ZwCwWFeL_H3ia26bP2e7HiKLWt0ZmGXPVwPO6uXq0vaA%40mail.gmail.com --- src/backend/access/heap/heapam.c | 33 +++++- src/backend/access/heap/heapam_handler.c | 125 +++++++++++++++------- src/backend/executor/nodeBitmapHeapscan.c | 98 +++-------------- src/include/access/heapam.h | 29 ++++- src/include/access/tableam.h | 18 +--- src/include/executor/nodeBitmapHeapscan.h | 7 ++ src/include/nodes/execnodes.h | 19 ---- 7 files changed, 168 insertions(+), 161 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index c35dc3c4e7..5728b2ea8a 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -975,8 +975,10 @@ heap_beginscan_bm(Relation relation, Snapshot snapshot, uint32 flags) scan->iterator.serial = NULL; scan->iterator.parallel = NULL; - scan->vmbuffer = InvalidBuffer; + scan->pf_iterator.serial = NULL; + scan->pf_iterator.parallel = NULL; + scan->pvmbuffer = InvalidBuffer; return (TableScanDesc) scan; } @@ -1081,6 +1083,10 @@ heap_endscan_bm(TableScanDesc sscan) if (BufferIsValid(scan->vmbuffer)) ReleaseBuffer(scan->vmbuffer); + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + + bhs_end_iterate(&scan->pf_iterator); bhs_end_iterate(&scan->iterator); /* @@ -1390,7 +1396,7 @@ heap_getnextslot_tidrange(TableScanDesc sscan, ScanDirection direction, void heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa) + ParallelBitmapHeapState *pstate, dsa_area *dsa, int pf_maximum) { BitmapHeapScanDesc scan = (BitmapHeapScanDesc) sscan; @@ -1401,9 +1407,19 @@ heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, ReleaseBuffer(scan->vmbuffer); scan->vmbuffer = InvalidBuffer; + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + scan->pvmbuffer = InvalidBuffer; + + bhs_end_iterate(&scan->pf_iterator); bhs_end_iterate(&scan->iterator); + scan->prefetch_maximum = 0; scan->empty_tuples_pending = 0; + scan->pstate = NULL; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + scan->pfblockno = InvalidBlockNumber; /* * reinitialize heap scan descriptor @@ -1414,6 +1430,19 @@ heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, pstate ? pstate->tbmiterator : InvalidDsaPointer); + +#ifdef USE_PREFETCH + if (pf_maximum > 0) + { + bhs_begin_iterate(&scan->pf_iterator, tbm, dsa, + pstate ? + pstate->prefetch_iterator : + InvalidDsaPointer); + } +#endif /* USE_PREFETCH */ + + scan->pstate = pstate; + scan->prefetch_maximum = pf_maximum; } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index ebaba33406..fca9c7233e 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -60,6 +60,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan); +static inline void BitmapPrefetch(BitmapHeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2187,26 +2190,26 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * iterator in prefetch_pages. For each block the main iterator returns, we * decrement prefetch_pages. */ -void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) +static inline void +BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = &node->pf_iterator; + BitmapHeapIterator *prefetch_iterator = &scan->pf_iterator; + ParallelBitmapHeapState *pstate = scan->pstate; TBMIterateResult *tbmpre; if (pstate == NULL) { - if (node->prefetch_pages > 0) + if (scan->prefetch_pages > 0) { /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; + scan->prefetch_pages--; } else if (!prefetch_iterator->exhausted) { /* Do not let the prefetch iterator get behind the main one */ tbmpre = bhs_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; } return; } @@ -2216,7 +2219,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) * table_scan_bitmap_next_block() keeps prefetch distance higher across * the parallel workers. */ - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { SpinLockAcquire(&pstate->mutex); if (pstate->prefetch_pages > 0) @@ -2240,7 +2243,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) if (!prefetch_iterator->exhausted) { tbmpre = bhs_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; } } } @@ -2256,33 +2259,34 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) * page/tuple, then to one after the second tuple is fetched, then * it doubles as later pages are fetched. */ -void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) +static inline void +BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; + ParallelBitmapHeapState *pstate = scan->pstate; + int prefetch_maximum = scan->prefetch_maximum; if (pstate == NULL) { - if (node->prefetch_target >= node->prefetch_maximum) + if (scan->prefetch_target >= prefetch_maximum) /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; + else if (scan->prefetch_target >= prefetch_maximum / 2) + scan->prefetch_target = prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; else - node->prefetch_target++; + scan->prefetch_target++; return; } /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) + if (pstate->prefetch_target < prefetch_maximum) { SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) + if (pstate->prefetch_target >= prefetch_maximum) /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; + else if (pstate->prefetch_target >= prefetch_maximum / 2) + pstate->prefetch_target = prefetch_maximum; else if (pstate->prefetch_target > 0) pstate->prefetch_target *= 2; else @@ -2295,18 +2299,19 @@ BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) /* * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target */ -void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) +static inline void +BitmapPrefetch(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - BitmapHeapIterator *prefetch_iterator = &node->pf_iterator; + ParallelBitmapHeapState *pstate = scan->pstate; + BitmapHeapIterator *prefetch_iterator = &scan->pf_iterator; + Relation rel = scan->heap_common.rs_base.rs_rd; if (pstate == NULL) { if (!prefetch_iterator->exhausted) { - while (node->prefetch_pages < node->prefetch_target) + while (scan->prefetch_pages < scan->prefetch_target) { TBMIterateResult *tbmpre = bhs_iterate(prefetch_iterator); bool skip_fetch; @@ -2317,8 +2322,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) bhs_end_iterate(prefetch_iterator); break; } - node->prefetch_pages++; - node->pfblockno = tbmpre->blockno; + scan->prefetch_pages++; + scan->pfblockno = tbmpre->blockno; /* * If we expect not to have to actually read this heap page, @@ -2326,14 +2331,14 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) * logic normally. (Would it be better not to increment * prefetch_pages?) */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLE) && !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, + VM_ALL_VISIBLE(rel, tbmpre->blockno, - &node->pvmbuffer)); + &scan->pvmbuffer)); if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); } } @@ -2373,17 +2378,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) break; } - node->pfblockno = tbmpre->blockno; + scan->pfblockno = tbmpre->blockno; /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLE) && + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLE) && !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, + VM_ALL_VISIBLE(rel, tbmpre->blockno, - &node->pvmbuffer)); + &scan->pvmbuffer)); if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); } } } @@ -2416,6 +2421,8 @@ heapam_scan_bitmap_next_block(TableScanDesc sscan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(scan); + do { CHECK_FOR_INTERRUPTS(); @@ -2556,6 +2563,18 @@ heapam_scan_bitmap_next_block(TableScanDesc sscan, else (*exact_pages)++; + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->pstate == NULL && + !scan->pf_iterator.exhausted && + scan->pfblockno < block) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(scan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2592,6 +2611,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc scan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!bscan->pstate) + { + if (bscan->prefetch_target < bscan->prefetch_maximum) + bscan->prefetch_target++; + } + else if (bscan->pstate->prefetch_target < bscan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&bscan->pstate->mutex); + if (bscan->pstate->prefetch_target < bscan->prefetch_maximum) + bscan->pstate->prefetch_target++; + SpinLockRelease(&bscan->pstate->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(bscan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 6f1afd427d..ae36dfd773 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -124,7 +124,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TIDBitmap *tbm; TupleTableSlot *slot; ParallelBitmapHeapState *pstate = node->pstate; - dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* * extract necessary information from index scan node @@ -148,7 +147,9 @@ BitmapHeapNext(BitmapHeapScanState *node) */ if (!node->initialized) { + Relation rel = node->ss.ss_currentRelation; uint32 extra_flags = 0; + bool pf_maximum = 0; /* * The leader will immediately come out of the function, but others @@ -157,6 +158,15 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ +#ifdef USE_PREFETCH + pf_maximum = get_tablespace_io_concurrency(rel->rd_rel->reltablespace); +#endif + + if (!pstate || init_shared_state) { tbm = (TIDBitmap *) MultiExecProcNode(outerPlanState(node)); @@ -174,7 +184,7 @@ BitmapHeapNext(BitmapHeapScanState *node) */ pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (pf_maximum > 0) { pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); @@ -200,25 +210,16 @@ BitmapHeapNext(BitmapHeapScanState *node) * scan descriptor and begin the scan. */ scan = table_beginscan_bm(node->ss.ss_currentScanDesc, - node->ss.ss_currentRelation, + rel, node->ss.ps.state->es_snapshot, extra_flags, + pf_maximum, node->tbm, node->pstate, node->ss.ps.state->es_query_dsa); node->ss.ss_currentScanDesc = scan; -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) - { - bhs_begin_iterate(&node->pf_iterator, tbm, dsa, - pstate ? - pstate->prefetch_iterator : - InvalidDsaPointer); - } -#endif /* USE_PREFETCH */ - node->initialized = true; goto new_page; @@ -230,37 +231,6 @@ BitmapHeapNext(BitmapHeapScanState *node) { CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); - /* * If we are using lossy info, we have to recheck the qual * conditions at every tuple. @@ -283,23 +253,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - if (!table_scan_bitmap_next_block(scan, &node->blockno, &node->recheck, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - !node->pf_iterator.exhausted && - node->pfblockno < node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -365,21 +321,12 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) PlanState *outerPlan = outerPlanState(node); /* release bitmaps and buffers if any */ - if (node->pf_iterator.exhausted) - bhs_end_iterate(&node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; ExecScanReScan(&node->ss); @@ -418,14 +365,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan_bm(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (!node->pf_iterator.exhausted) - bhs_end_iterate(&node->pf_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -458,16 +401,12 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = -1; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -507,13 +446,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index f81a295685..499e739da7 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -93,6 +93,10 @@ typedef struct BitmapHeapScanDescData * Members common to Parallel and Serial BitmapHeapScan */ BitmapHeapIterator iterator; + BitmapHeapIterator pf_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; /* * These fields are only used for bitmap scans for the "skip fetch" @@ -102,7 +106,26 @@ typedef struct BitmapHeapScanDescData * to return. They are common to parallel and serial BitmapHeapScans */ Buffer vmbuffer; + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; int empty_tuples_pending; + + /* + * Parallel-only members + */ + + struct ParallelBitmapHeapState *pstate; + + /* + * Serial-only members + */ + + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; } BitmapHeapScanDescData; typedef struct BitmapHeapScanDescData *BitmapHeapScanDesc; @@ -307,7 +330,7 @@ extern TableScanDesc heap_beginscan_bm(Relation relation, Snapshot snapshot, uin extern void heap_endscan_bm(TableScanDesc scan); extern void heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa); + ParallelBitmapHeapState *pstate, dsa_area *dsa, int pf_maximum); extern bool heap_fetch(Relation relation, Snapshot snapshot, HeapTuple tuple, Buffer *userbuf, bool keep_buf); @@ -426,10 +449,6 @@ extern bool heapam_scan_analyze_next_tuple(TableScanDesc scan, double *liverows, double *deadrows, TupleTableSlot *slot); -extern void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -extern void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -extern void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); - /* * To avoid leaking too much knowledge about reorderbuffer implementation * details this is implemented in reorderbuffer.c not heapam_visibility.c diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index f344d97f1f..9ec0a1da2b 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -399,7 +399,8 @@ typedef struct TableAmRoutine uint32 flags); void (*scan_rescan_bm) (TableScanDesc scan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa); + ParallelBitmapHeapState *pstate, dsa_area *dsa, + int pf_maximum); /* * Release resources and deallocate scan. If TableScanDesc.temp_snap, @@ -801,17 +802,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the bitmap is lossy for the selected * block; otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -950,7 +940,7 @@ table_beginscan_strat(Relation rel, Snapshot snapshot, */ static inline TableScanDesc table_beginscan_bm(TableScanDesc scan, Relation rel, Snapshot snapshot, - uint32 extra_flags, TIDBitmap *tbm, + uint32 extra_flags, int pf_maximum, TIDBitmap *tbm, ParallelBitmapHeapState *pstate, dsa_area *dsa) { uint32 flags = SO_TYPE_BITMAPSCAN | SO_ALLOW_PAGEMODE | extra_flags; @@ -962,7 +952,7 @@ table_beginscan_bm(TableScanDesc scan, Relation rel, Snapshot snapshot, if (!scan) scan = rel->rd_tableam->scan_begin_bm(rel, snapshot, flags); - scan->rs_rd->rd_tableam->scan_rescan_bm(scan, tbm, pstate, dsa); + scan->rs_rd->rd_tableam->scan_rescan_bm(scan, tbm, pstate, dsa, pf_maximum); return scan; } diff --git a/src/include/executor/nodeBitmapHeapscan.h b/src/include/executor/nodeBitmapHeapscan.h index 7064f54686..028081db32 100644 --- a/src/include/executor/nodeBitmapHeapscan.h +++ b/src/include/executor/nodeBitmapHeapscan.h @@ -29,6 +29,13 @@ extern void ExecBitmapHeapReInitializeDSM(BitmapHeapScanState *node, extern void ExecBitmapHeapInitializeWorker(BitmapHeapScanState *node, ParallelWorkerContext *pwcxt); +typedef struct BitmapHeapIterator +{ + struct TBMIterator *serial; + struct TBMSharedIterator *parallel; + bool exhausted; +} BitmapHeapIterator; + extern TBMIterateResult *bhs_iterate(BitmapHeapIterator *iterator); extern void bhs_begin_iterate(BitmapHeapIterator *iterator, TIDBitmap *tbm, dsa_area *dsa, dsa_pointer dsp); diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 714eb3e534..713de7d50f 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1787,30 +1787,17 @@ typedef struct ParallelBitmapHeapState ConditionVariable cv; } ParallelBitmapHeapState; -typedef struct BitmapHeapIterator -{ - struct TBMIterator *serial; - struct TBMSharedIterator *parallel; - bool exhausted; -} BitmapHeapIterator; - /* ---------------- * BitmapHeapScanState information * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * pf_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1818,18 +1805,12 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - BitmapHeapIterator pf_iterator; ParallelBitmapHeapState *pstate; bool recheck; BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --e3xl7h75mzefinno Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v16-0017-Move-BitmapHeapScan-initialization-to-helper.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
* [PATCH v19 19/21] Push BitmapHeapScan prefetch code into heap AM @ 2024-04-06 20:21 Melanie Plageman <[email protected]> 0 siblings, 0 replies; 10+ messages in thread From: Melanie Plageman @ 2024-04-06 20:21 UTC (permalink / raw) An earlier commit eliminated a table AM layering violation for the current block in a BitmapHeapScan but left the violation in BitmapHeapScan prefetching code. To resolve this, move and manage all state used for prefetching entirely into the heap AM. Author: Melanie Plageman Discussion: https://postgr.es/m/CAAKRu_ZwCwWFeL_H3ia26bP2e7HiKLWt0ZmGXPVwPO6uXq0vaA%40mail.gmail.com --- src/backend/access/heap/heapam.c | 38 ++++++- src/backend/access/heap/heapam_handler.c | 125 +++++++++++++++------- src/backend/executor/nodeBitmapHeapscan.c | 97 +++-------------- src/include/access/heapam.h | 29 ++++- src/include/access/tableam.h | 18 +--- src/include/nodes/execnodes.h | 12 --- 6 files changed, 167 insertions(+), 152 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index a45fdb50285..71f93bfbb1d 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -1065,6 +1065,14 @@ heap_beginscan_bm(Relation relation, Snapshot snapshot, uint32 flags) scan->vmbuffer = InvalidBuffer; scan->empty_tuples_pending = 0; + /* + * The rest of the BitmapHeapScanDesc members related to prefetching will + * be initialized in heap_rescan_bm(). + */ + scan->pvmbuffer = InvalidBuffer; + scan->prefetch_iterator.serial = NULL; + scan->prefetch_iterator.parallel = NULL; + return (TableScanDesc) scan; } @@ -1108,7 +1116,8 @@ heap_rescan(TableScanDesc sscan, ScanKey key, bool set_params, void heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa) + ParallelBitmapHeapState *pstate, dsa_area *dsa, + int prefetch_maximum) { BitmapHeapScanDesc scan = (BitmapHeapScanDesc) sscan; @@ -1122,6 +1131,17 @@ heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, scan->empty_tuples_pending = 0; unified_tbm_end_iterate(&scan->iterator); + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + + unified_tbm_end_iterate(&scan->prefetch_iterator); + + scan->prefetch_maximum = prefetch_maximum; + scan->pstate = pstate; + scan->prefetch_target = -1; + scan->prefetch_pages = 0; + scan->pfblockno = InvalidBlockNumber; + /* * reinitialize heap scan descriptor */ @@ -1131,6 +1151,17 @@ heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, pstate ? pstate->tbmiterator : InvalidDsaPointer); + +#ifdef USE_PREFETCH + if (prefetch_maximum > 0) + { + unified_tbm_begin_iterate(&scan->prefetch_iterator, tbm, dsa, + pstate ? + pstate->prefetch_iterator : + InvalidDsaPointer); + } +#endif /* USE_PREFETCH */ + } @@ -1183,6 +1214,11 @@ heap_endscan_bm(TableScanDesc sscan) unified_tbm_end_iterate(&scan->iterator); + if (BufferIsValid(scan->pvmbuffer)) + ReleaseBuffer(scan->pvmbuffer); + + unified_tbm_end_iterate(&scan->prefetch_iterator); + /* * decrement relation reference count and free scan descriptor storage */ diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index 85564fef2e0..77888cdc90d 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -60,6 +60,9 @@ static bool SampleHeapTupleVisible(TableScanDesc scan, Buffer buffer, OffsetNumber tupoffset); static BlockNumber heapam_scan_get_blocks_done(HeapScanDesc hscan); +static inline void BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan); +static inline void BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan); +static inline void BitmapPrefetch(BitmapHeapScanDesc scan); static const TableAmRoutine heapam_methods; @@ -2187,26 +2190,26 @@ heapam_estimate_rel_size(Relation rel, int32 *attr_widths, * iterator in prefetch_pages. For each block the main iterator returns, we * decrement prefetch_pages. */ -void -BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) +static inline void +BitmapAdjustPrefetchIterator(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - UnifiedTBMIterator *prefetch_iterator = &node->prefetch_iterator; + ParallelBitmapHeapState *pstate = scan->pstate; + UnifiedTBMIterator *prefetch_iterator = &scan->prefetch_iterator; TBMIterateResult *tbmpre; if (pstate == NULL) { - if (node->prefetch_pages > 0) + if (scan->prefetch_pages > 0) { /* The main iterator has closed the distance by one page */ - node->prefetch_pages--; + scan->prefetch_pages--; } else if (!prefetch_iterator->exhausted) { /* Do not let the prefetch iterator get behind the main one */ tbmpre = unified_tbm_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; } return; } @@ -2220,7 +2223,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) * Note that moving the call site of BitmapAdjustPrefetchIterator() * exacerbates the effects of this bug. */ - if (node->prefetch_maximum > 0) + if (scan->prefetch_maximum > 0) { SpinLockAcquire(&pstate->mutex); if (pstate->prefetch_pages > 0) @@ -2244,7 +2247,7 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) if (!prefetch_iterator->exhausted) { tbmpre = unified_tbm_iterate(prefetch_iterator); - node->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; + scan->pfblockno = tbmpre ? tbmpre->blockno : InvalidBlockNumber; } } } @@ -2259,33 +2262,33 @@ BitmapAdjustPrefetchIterator(BitmapHeapScanState *node) * page/tuple, then to one after the second tuple is fetched, then * it doubles as later pages are fetched. */ -void -BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) +static inline void +BitmapAdjustPrefetchTarget(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; + ParallelBitmapHeapState *pstate = scan->pstate; if (pstate == NULL) { - if (node->prefetch_target >= node->prefetch_maximum) + if (scan->prefetch_target >= scan->prefetch_maximum) /* don't increase any further */ ; - else if (node->prefetch_target >= node->prefetch_maximum / 2) - node->prefetch_target = node->prefetch_maximum; - else if (node->prefetch_target > 0) - node->prefetch_target *= 2; + else if (scan->prefetch_target >= scan->prefetch_maximum / 2) + scan->prefetch_target = scan->prefetch_maximum; + else if (scan->prefetch_target > 0) + scan->prefetch_target *= 2; else - node->prefetch_target++; + scan->prefetch_target++; return; } /* Do an unlocked check first to save spinlock acquisitions. */ - if (pstate->prefetch_target < node->prefetch_maximum) + if (pstate->prefetch_target < scan->prefetch_maximum) { SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target >= node->prefetch_maximum) + if (pstate->prefetch_target >= scan->prefetch_maximum) /* don't increase any further */ ; - else if (pstate->prefetch_target >= node->prefetch_maximum / 2) - pstate->prefetch_target = node->prefetch_maximum; + else if (pstate->prefetch_target >= scan->prefetch_maximum / 2) + pstate->prefetch_target = scan->prefetch_maximum; else if (pstate->prefetch_target > 0) pstate->prefetch_target *= 2; else @@ -2298,18 +2301,19 @@ BitmapAdjustPrefetchTarget(BitmapHeapScanState *node) /* * BitmapPrefetch - Prefetch, if prefetch_pages are behind prefetch_target */ -void -BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) +static inline void +BitmapPrefetch(BitmapHeapScanDesc scan) { #ifdef USE_PREFETCH - ParallelBitmapHeapState *pstate = node->pstate; - UnifiedTBMIterator *prefetch_iterator = &node->prefetch_iterator; + ParallelBitmapHeapState *pstate = scan->pstate; + Relation rel = scan->heap_common.rs_base.rs_rd; + UnifiedTBMIterator *prefetch_iterator = &scan->prefetch_iterator; if (pstate == NULL) { if (!prefetch_iterator->exhausted) { - while (node->prefetch_pages < node->prefetch_target) + while (scan->prefetch_pages < scan->prefetch_target) { TBMIterateResult *tbmpre = unified_tbm_iterate(prefetch_iterator); bool skip_fetch; @@ -2320,8 +2324,8 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) unified_tbm_end_iterate(prefetch_iterator); break; } - node->prefetch_pages++; - node->pfblockno = tbmpre->blockno; + scan->prefetch_pages++; + scan->pfblockno = tbmpre->blockno; /* * If we expect not to have to actually read this heap page, @@ -2329,14 +2333,14 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) * logic normally. (Would it be better not to increment * prefetch_pages?) */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLES) && + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLES) && !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, + VM_ALL_VISIBLE(rel, tbmpre->blockno, - &node->pvmbuffer)); + &scan->pvmbuffer)); if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); } } @@ -2376,17 +2380,17 @@ BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan) break; } - node->pfblockno = tbmpre->blockno; + scan->pfblockno = tbmpre->blockno; /* As above, skip prefetch if we expect not to need page */ - skip_fetch = (!(scan->rs_flags & SO_NEED_TUPLES) && + skip_fetch = (!(scan->heap_common.rs_base.rs_flags & SO_NEED_TUPLES) && !tbmpre->recheck && - VM_ALL_VISIBLE(node->ss.ss_currentRelation, + VM_ALL_VISIBLE(rel, tbmpre->blockno, - &node->pvmbuffer)); + &scan->pvmbuffer)); if (!skip_fetch) - PrefetchBuffer(scan->rs_rd, MAIN_FORKNUM, tbmpre->blockno); + PrefetchBuffer(rel, MAIN_FORKNUM, tbmpre->blockno); } } } @@ -2419,6 +2423,8 @@ heapam_scan_bitmap_next_block(TableScanDesc sscan, *blockno = InvalidBlockNumber; *recheck = true; + BitmapAdjustPrefetchIterator(scan); + do { CHECK_FOR_INTERRUPTS(); @@ -2559,6 +2565,19 @@ heapam_scan_bitmap_next_block(TableScanDesc sscan, else (*exact_pages)++; + + /* + * If serial, we can error out if the the prefetch block doesn't stay + * ahead of the current block. + */ + if (scan->pstate == NULL && + !scan->prefetch_iterator.exhausted && + scan->pfblockno < block) + elog(ERROR, "prefetch and main iterators are out of sync"); + + /* Adjust the prefetch target */ + BitmapAdjustPrefetchTarget(scan); + /* * Return true to indicate that a valid block was found and the bitmap is * not exhausted. If there are no visible tuples on this page, @@ -2595,6 +2614,36 @@ heapam_scan_bitmap_next_tuple(TableScanDesc sscan, if (hscan->rs_cindex < 0 || hscan->rs_cindex >= hscan->rs_ntuples) return false; +#ifdef USE_PREFETCH + + /* + * Try to prefetch at least a few pages even before we get to the second + * page if we don't stop reading after the first tuple. + */ + if (!scan->pstate) + { + if (scan->prefetch_target < scan->prefetch_maximum) + scan->prefetch_target++; + } + else if (scan->pstate->prefetch_target < scan->prefetch_maximum) + { + /* take spinlock while updating shared state */ + SpinLockAcquire(&scan->pstate->mutex); + if (scan->pstate->prefetch_target < scan->prefetch_maximum) + scan->pstate->prefetch_target++; + SpinLockRelease(&scan->pstate->mutex); + } + + /* + * We issue prefetch requests *after* fetching the current page to try to + * avoid having prefetching interfere with the main I/O. Also, this should + * happen only when we have determined there is still something to do on + * the current page, else we may uselessly prefetch the same page we are + * just about to request for real. + */ + BitmapPrefetch(scan); +#endif /* USE_PREFETCH */ + targoffset = hscan->rs_vistuples[hscan->rs_cindex]; page = BufferGetPage(hscan->rs_cbuf); lp = PageGetItemId(page, targoffset); diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index fda5645e4ae..11945ab9c42 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -69,7 +69,6 @@ BitmapHeapNext(BitmapHeapScanState *node) TIDBitmap *tbm; TupleTableSlot *slot; ParallelBitmapHeapState *pstate = node->pstate; - dsa_area *dsa = node->ss.ps.state->es_query_dsa; /* * extract necessary information from index scan node @@ -93,7 +92,9 @@ BitmapHeapNext(BitmapHeapScanState *node) */ if (!node->initialized) { + Relation rel = node->ss.ss_currentRelation; bool need_tuples = false; + int prefetch_maximum = 0; /* * The leader will immediately come out of the function, but others @@ -103,6 +104,14 @@ BitmapHeapNext(BitmapHeapScanState *node) bool init_shared_state = node->pstate ? BitmapShouldInitializeSharedState(node->pstate) : false; + /* + * Maximum number of prefetches for the tablespace if configured, + * otherwise the current value of the effective_io_concurrency GUC. + */ +#ifdef USE_PREFETCH + prefetch_maximum = get_tablespace_io_concurrency(rel->rd_rel->reltablespace); +#endif + /* * Only serial bitmap table scans and the parallel leader in a * parallel bitmap table scan should build the bitmap. @@ -124,7 +133,7 @@ BitmapHeapNext(BitmapHeapScanState *node) */ pstate->tbmiterator = tbm_prepare_shared_iterate(tbm); #ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) + if (prefetch_maximum > 0) { pstate->prefetch_iterator = tbm_prepare_shared_iterate(tbm); @@ -154,22 +163,13 @@ BitmapHeapNext(BitmapHeapScanState *node) node->ss.ss_currentRelation, node->ss.ps.state->es_snapshot, need_tuples, + prefetch_maximum, node->tbm, node->pstate, node->ss.ps.state->es_query_dsa); node->ss.ss_currentScanDesc = scan; -#ifdef USE_PREFETCH - if (node->prefetch_maximum > 0) - { - unified_tbm_begin_iterate(&node->prefetch_iterator, tbm, dsa, - pstate ? - pstate->prefetch_iterator : - InvalidDsaPointer); - } -#endif /* USE_PREFETCH */ - node->initialized = true; goto new_page; @@ -185,37 +185,6 @@ BitmapHeapNext(BitmapHeapScanState *node) CHECK_FOR_INTERRUPTS(); -#ifdef USE_PREFETCH - - /* - * Try to prefetch at least a few pages even before we get to the - * second page if we don't stop reading after the first tuple. - */ - if (!pstate) - { - if (node->prefetch_target < node->prefetch_maximum) - node->prefetch_target++; - } - else if (pstate->prefetch_target < node->prefetch_maximum) - { - /* take spinlock while updating shared state */ - SpinLockAcquire(&pstate->mutex); - if (pstate->prefetch_target < node->prefetch_maximum) - pstate->prefetch_target++; - SpinLockRelease(&pstate->mutex); - } -#endif /* USE_PREFETCH */ - - /* - * We issue prefetch requests *after* fetching the current page to - * try to avoid having prefetching interfere with the main I/O. - * Also, this should happen only when we have determined there is - * still something to do on the current page, else we may - * uselessly prefetch the same page we are just about to request - * for real. - */ - BitmapPrefetch(node, scan); - /* * If we are using lossy info, we have to recheck the qual * conditions at every tuple. @@ -238,23 +207,9 @@ BitmapHeapNext(BitmapHeapScanState *node) new_page: - BitmapAdjustPrefetchIterator(node); - if (!table_scan_bitmap_next_block(scan, &node->blockno, &node->recheck, &node->lossy_pages, &node->exact_pages)) break; - - /* - * If serial, we can error out if the the prefetch block doesn't stay - * ahead of the current block. - */ - if (node->pstate == NULL && - !node->prefetch_iterator.exhausted && - node->pfblockno < node->blockno) - elog(ERROR, "prefetch and main iterators are out of sync"); - - /* Adjust the prefetch target */ - BitmapAdjustPrefetchTarget(node); } /* @@ -319,22 +274,13 @@ ExecReScanBitmapHeapScan(BitmapHeapScanState *node) { PlanState *outerPlan = outerPlanState(node); - /* release bitmaps and buffers if any */ - if (node->prefetch_iterator.exhausted) - unified_tbm_end_iterate(&node->prefetch_iterator); + /* release bitmaps if any */ if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); node->tbm = NULL; node->initialized = false; - node->pvmbuffer = InvalidBuffer; node->recheck = true; node->blockno = InvalidBlockNumber; - node->pfblockno = InvalidBlockNumber; - /* Only used for serial BHS */ - node->prefetch_pages = 0; - node->prefetch_target = -1; ExecScanReScan(&node->ss); @@ -373,14 +319,10 @@ ExecEndBitmapHeapScan(BitmapHeapScanState *node) table_endscan_bm(scanDesc); /* - * release bitmaps and buffers if any + * release bitmaps if any */ - if (!node->prefetch_iterator.exhausted) - unified_tbm_end_iterate(&node->prefetch_iterator); if (node->tbm) tbm_free(node->tbm); - if (node->pvmbuffer != InvalidBuffer) - ReleaseBuffer(node->pvmbuffer); } /* ---------------------------------------------------------------- @@ -413,16 +355,12 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->ss.ps.ExecProcNode = ExecBitmapHeapScan; scanstate->tbm = NULL; - scanstate->pvmbuffer = InvalidBuffer; scanstate->exact_pages = 0; scanstate->lossy_pages = 0; - scanstate->prefetch_pages = 0; - scanstate->prefetch_target = -1; scanstate->initialized = false; scanstate->pstate = NULL; scanstate->recheck = true; scanstate->blockno = InvalidBlockNumber; - scanstate->pfblockno = InvalidBlockNumber; /* * Miscellaneous initialization @@ -462,13 +400,6 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); - /* - * Maximum number of prefetches for the tablespace if configured, - * otherwise the current value of the effective_io_concurrency GUC. - */ - scanstate->prefetch_maximum = - get_tablespace_io_concurrency(currentRelation->rd_rel->reltablespace); - scanstate->ss.ss_currentRelation = currentRelation; /* diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index f0cd0041460..b17f0e8b4e6 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -92,6 +92,10 @@ typedef struct BitmapHeapScanDescData * Members common to Parallel and Serial BitmapHeapScan */ UnifiedTBMIterator iterator; + UnifiedTBMIterator prefetch_iterator; + + /* maximum value for prefetch_target */ + int prefetch_maximum; /* * These fields are only used for bitmap scans for the "skip fetch" @@ -101,7 +105,26 @@ typedef struct BitmapHeapScanDescData * to return. They are common to parallel and serial BitmapHeapScans */ Buffer vmbuffer; + /* buffer for visibility-map lookups of prefetched pages */ + Buffer pvmbuffer; int empty_tuples_pending; + + /* + * Parallel-only members + */ + + struct ParallelBitmapHeapState *pstate; + + /* + * Serial-only members + */ + + /* Current target for prefetch distance */ + int prefetch_target; + /* # pages prefetch iterator is ahead of current */ + int prefetch_pages; + /* used to validate prefetch block stays ahead of current block */ + BlockNumber pfblockno; } BitmapHeapScanDescData; typedef struct BitmapHeapScanDescData *BitmapHeapScanDesc; @@ -305,7 +328,8 @@ extern bool heap_getnextslot_tidrange(TableScanDesc sscan, extern TableScanDesc heap_beginscan_bm(Relation relation, Snapshot snapshot, uint32 flags); extern void heap_endscan_bm(TableScanDesc scan); extern void heap_rescan_bm(TableScanDesc sscan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa); + ParallelBitmapHeapState *pstate, dsa_area *dsa, + int prefetch_maximum); extern bool heap_fetch(Relation relation, Snapshot snapshot, HeapTuple tuple, Buffer *userbuf, bool keep_buf); extern bool heap_hot_search_buffer(ItemPointer tid, Relation relation, @@ -422,9 +446,6 @@ extern bool heapam_scan_analyze_next_tuple(TableScanDesc scan, TransactionId OldestXmin, double *liverows, double *deadrows, TupleTableSlot *slot); -extern void BitmapAdjustPrefetchIterator(BitmapHeapScanState *node); -extern void BitmapAdjustPrefetchTarget(BitmapHeapScanState *node); -extern void BitmapPrefetch(BitmapHeapScanState *node, TableScanDesc scan); /* * To avoid leaking too much knowledge about reorderbuffer implementation diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index aebcecad30d..06530536897 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -368,7 +368,8 @@ typedef struct TableAmRoutine uint32 flags); void (*scan_rescan_bm) (TableScanDesc scan, TIDBitmap *tbm, - ParallelBitmapHeapState *pstate, dsa_area *dsa); + ParallelBitmapHeapState *pstate, dsa_area *dsa, + int prefetch_maximum); /* * Release resources and deallocate scan. @@ -805,17 +806,6 @@ typedef struct TableAmRoutine * lossy_pages is incremented if the bitmap is lossy for the selected * block; otherwise, exact_pages is incremented. * - * XXX: Currently this may only be implemented if the AM uses md.c as its - * storage manager, and uses ItemPointer->ip_blkid in a manner that maps - * blockids directly to the underlying storage. nodeBitmapHeapscan.c - * performs prefetching directly using that interface. This probably - * needs to be rectified at a later point. - * - * XXX: Currently this may only be implemented if the AM uses the - * visibilitymap, as nodeBitmapHeapscan.c unconditionally accesses it to - * perform prefetching. This probably needs to be rectified at a later - * point. - * * Optional callback, but either both scan_bitmap_next_block and * scan_bitmap_next_tuple need to exist, or neither. */ @@ -954,7 +944,7 @@ table_beginscan_strat(Relation rel, Snapshot snapshot, */ static inline TableScanDesc table_beginscan_bm(TableScanDesc scan, Relation rel, Snapshot snapshot, - bool need_tuple, TIDBitmap *tbm, + bool need_tuple, int prefetch_maximum, TIDBitmap *tbm, ParallelBitmapHeapState *pstate, dsa_area *dsa) { uint32 flags = SO_TYPE_BITMAPSCAN | SO_ALLOW_PAGEMODE; @@ -969,7 +959,7 @@ table_beginscan_bm(TableScanDesc scan, Relation rel, Snapshot snapshot, if (!scan) scan = rel->rd_tableam->scan_begin_bm(rel, snapshot, flags); - scan->rs_rd->rd_tableam->scan_rescan_bm(scan, tbm, pstate, dsa); + scan->rs_rd->rd_tableam->scan_rescan_bm(scan, tbm, pstate, dsa, prefetch_maximum); return scan; } diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index 6693cf66c77..713de7d50f6 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1792,18 +1792,12 @@ typedef struct ParallelBitmapHeapState * * bitmapqualorig execution state for bitmapqualorig expressions * tbm bitmap obtained from child index scan(s) - * pvmbuffer buffer for visibility-map lookups of prefetched pages * exact_pages total number of exact pages retrieved * lossy_pages total number of lossy pages retrieved - * prefetch_iterator for prefetching ahead of current page - * prefetch_pages # pages prefetch iterator is ahead of current - * prefetch_target current target prefetch distance - * prefetch_maximum maximum value for prefetch_target * initialized is node is ready to iterate * pstate shared state for parallel bitmap scan * recheck do current page's tuples need recheck * blockno used to validate pf and current block in sync - * pfblockno used to validate pf stays ahead of current block * ---------------- */ typedef struct BitmapHeapScanState @@ -1811,18 +1805,12 @@ typedef struct BitmapHeapScanState ScanState ss; /* its first field is NodeTag */ ExprState *bitmapqualorig; TIDBitmap *tbm; - Buffer pvmbuffer; long exact_pages; long lossy_pages; - int prefetch_pages; - int prefetch_target; - int prefetch_maximum; bool initialized; - UnifiedTBMIterator prefetch_iterator; ParallelBitmapHeapState *pstate; bool recheck; BlockNumber blockno; - BlockNumber pfblockno; } BitmapHeapScanState; /* ---------------- -- 2.40.1 --clv5reqgj4vdihqz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v19-0020-Move-BitmapHeapScan-initialization-to-helper.patch" ^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2024-04-06 20:21 UTC | newest] Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-08-03 23:21 [PATCH v5 2/2] ALTER TABLE ... DETACH CONCURRENTLY Alvaro Herrera <[email protected]> 2024-03-22 13:42 [PATCH v12 14/17] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v8 13/17] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v13 14/16] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v15 11/13] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v10 14/17] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v11 14/17] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-03-22 13:42 [PATCH v9 13/17] Push BitmapHeapScan prefetch code into heapam.c Melanie Plageman <[email protected]> 2024-04-05 22:48 [PATCH v16 16/18] Push BitmapHeapScan prefetch code into heap AM Melanie Plageman <[email protected]> 2024-04-06 20:21 [PATCH v19 19/21] Push BitmapHeapScan prefetch code into heap AM Melanie Plageman <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox