agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v10 2/2] handle relation statistics correctly during rewrites 7+ messages / 2 participants [nested] [flat]
* [PATCH v10 2/2] handle relation statistics correctly during rewrites @ 2025-11-04 13:52 Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Bertrand Drouvot @ 2025-11-04 13:52 UTC (permalink / raw) Now that PGSTAT_KIND_RELATION is keyed by refilenode, we need to handle rewrites. To do so, this patch: - Adds PgStat_PendingRewrite, a new struct to track rewrite operations within a transaction, storing the old locator, new locator, and original locator (for rewrite chains). This allows stats to be copied from the original location to the final location at commit time. - Adds a new function, pgstat_mark_rewrite(), called when a table rewrite begins. It records the rewrite operation in a local list and detects rewrite chains by checking if the old_locator matches any existing new_locator, preserving the chain's original_locator. - Modifies pgstat_copy_relation_stats(), to accept RelFileLocators instead of Relations, with a new increment parameter to accumulate stats (needed for rewrite chains with DML between rewrites). - Ensures that AtEOXact_PgStat_Relations(), AtPrepare_PgStat_Relations(), pgstat_twophase_postcommit()/postabort() pgstat_drop_relation() handle the PgStat_PendingRewrite list correctly. Note that due to the new flush call in pgstat_twophase_postcommit() we can not call GetCurrentTransactionStopTimestamp() in pgstat_relation_flush_cb(). So, adding a check to handle this special case and call GetCurrentTimestamp() instead. Note that we'd call GetCurrentTimestamp() only if there is a rewrite, so that the GetCurrentTimestamp() extra cost should be negligible. Another solution could be to trigger the flush from FinishPreparedTransaction() but that's not worth the extra complexity. The new pending_rewrites list is traversed in multiple places. The overhead should be negligible in comparison to a rewrite and the list should not contain a lot of rewrites in practice. The pending_rewrites list is traversed in multiple places. In typical usage, the list will contain only a few entries so the traversal cost is negligible ( furthermore in comparison to a rewrite). --- src/backend/catalog/index.c | 2 +- src/backend/commands/cluster.c | 5 + src/backend/commands/tablecmds.c | 6 + src/backend/utils/activity/pgstat_relation.c | 391 ++++++++++++++++++- src/backend/utils/activity/pgstat_xact.c | 25 +- src/backend/utils/cache/relcache.c | 6 + src/include/pgstat.h | 5 +- src/tools/pgindent/typedefs.list | 1 + 8 files changed, 424 insertions(+), 17 deletions(-) 92.8% src/backend/utils/activity/ 4.9% src/backend/ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 8dea58ad96b..b71925a22c3 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1795,7 +1795,7 @@ index_concurrently_swap(Oid newIndexId, Oid oldIndexId, const char *oldName) changeDependenciesOn(RelationRelationId, oldIndexId, newIndexId); /* copy over statistics from old to new index */ - pgstat_copy_relation_stats(newClassRel, oldClassRel); + pgstat_copy_relation_stats(newClassRel->rd_locator, oldClassRel->rd_locator, false); /* Copy data of pg_statistic from the old index to the new one */ CopyStatistics(oldIndexId, newIndexId); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2120c85ccb4..6155b12afab 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1196,6 +1196,11 @@ swap_relation_files(Oid r1, Oid r2, bool target_is_pg_class, rel1 = relation_open(r1, NoLock); rel2 = relation_open(r2, NoLock); + + /* Mark that a rewrite happened */ + if (RELKIND_HAS_STORAGE(rel1->rd_rel->relkind)) + pgstat_mark_rewrite(rel1->rd_locator, rel2->rd_locator); + rel2->rd_createSubid = rel1->rd_createSubid; rel2->rd_newRelfilelocatorSubid = rel1->rd_newRelfilelocatorSubid; rel2->rd_firstRelfilelocatorSubid = rel1->rd_firstRelfilelocatorSubid; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 6b1a00ed477..9de70f321ed 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -16884,6 +16884,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode) Oid reltoastrelid; RelFileNumber newrelfilenumber; RelFileLocator newrlocator; + RelFileLocator oldrlocator; List *reltoastidxids = NIL; ListCell *lc; @@ -16922,6 +16923,7 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode) newrlocator = rel->rd_locator; newrlocator.relNumber = newrelfilenumber; newrlocator.spcOid = newTableSpace; + oldrlocator = rel->rd_locator; /* hand off to AM to actually create new rel storage and copy the data */ if (rel->rd_rel->relkind == RELKIND_INDEX) @@ -16934,6 +16936,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode) table_relation_copy_data(rel, &newrlocator); } + /* mark that a rewrite happened */ + if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind)) + pgstat_mark_rewrite(oldrlocator, newrlocator); + /* * Update the pg_class row. * diff --git a/src/backend/utils/activity/pgstat_relation.c b/src/backend/utils/activity/pgstat_relation.c index a267e93f8be..123fb50d98f 100644 --- a/src/backend/utils/activity/pgstat_relation.c +++ b/src/backend/utils/activity/pgstat_relation.c @@ -30,6 +30,19 @@ #include "utils/syscache.h" #include "utils/timestamp.h" +/* Pending rewrite operations for stats copying */ +typedef struct PgStat_PendingRewrite +{ + RelFileLocator old_locator; + RelFileLocator new_locator; + RelFileLocator original_locator; + int nest_level; /* Transaction nesting level where rewrite + * occurred */ + struct PgStat_PendingRewrite *next; +} PgStat_PendingRewrite; + +/* The pending rewrites list for current transaction */ +static PgStat_PendingRewrite *pending_rewrites = NULL; /* Record that's written to 2PC state file when pgstat state is persisted */ typedef struct TwoPhasePgStatRecord @@ -43,6 +56,8 @@ typedef struct TwoPhasePgStatRecord PgStat_Counter deleted_pre_truncdrop; RelFileLocator locator; /* table's rd_locator */ bool truncdropped; /* was the relation truncated/dropped? */ + RelFileLocator rewrite_old_locator; + int rewrite_nest_level; } TwoPhasePgStatRecord; @@ -54,27 +69,70 @@ static void restore_truncdrop_counters(PgStat_TableXactStatus *trans); /* - * Copy stats between relations. This is used for things like REINDEX + * Copy stats between RelFileLocator. This is used for things like REINDEX * CONCURRENTLY. */ void -pgstat_copy_relation_stats(Relation dst, Relation src) +pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment) { PgStat_StatTabEntry *srcstats; PgStatShared_Relation *dstshstats; PgStat_EntryRef *dst_ref; - srcstats = pgstat_fetch_stat_tabentry_ext(RelationGetRelid(src)); + srcstats = (PgStat_StatTabEntry *) pgstat_fetch_entry(PGSTAT_KIND_RELATION, + src.dbOid, + RelFileLocatorToPgStatObjid(src)); if (!srcstats) return; dst_ref = pgstat_get_entry_ref_locked(PGSTAT_KIND_RELATION, - dst->rd_rel->relisshared ? InvalidOid : MyDatabaseId, - RelationGetRelid(dst), + dst.dbOid, + RelFileLocatorToPgStatObjid(dst), false); dstshstats = (PgStatShared_Relation *) dst_ref->shared_stats; - dstshstats->stats = *srcstats; + + if (!increment) + dstshstats->stats = *srcstats; + else + { + /* Increment those statistics */ +#define RELFSTAT_ACC(fld, stats_to_add) \ + (dstshstats->stats.fld += stats_to_add->fld) + RELFSTAT_ACC(numscans, srcstats); + RELFSTAT_ACC(tuples_returned, srcstats); + RELFSTAT_ACC(tuples_fetched, srcstats); + RELFSTAT_ACC(tuples_inserted, srcstats); + RELFSTAT_ACC(tuples_updated, srcstats); + RELFSTAT_ACC(tuples_deleted, srcstats); + RELFSTAT_ACC(tuples_hot_updated, srcstats); + RELFSTAT_ACC(tuples_newpage_updated, srcstats); + RELFSTAT_ACC(live_tuples, srcstats); + RELFSTAT_ACC(dead_tuples, srcstats); + RELFSTAT_ACC(mod_since_analyze, srcstats); + RELFSTAT_ACC(ins_since_vacuum, srcstats); + RELFSTAT_ACC(blocks_fetched, srcstats); + RELFSTAT_ACC(blocks_hit, srcstats); + RELFSTAT_ACC(vacuum_count, srcstats); + RELFSTAT_ACC(autovacuum_count, srcstats); + RELFSTAT_ACC(analyze_count, srcstats); + RELFSTAT_ACC(autoanalyze_count, srcstats); + RELFSTAT_ACC(total_vacuum_time, srcstats); + RELFSTAT_ACC(total_autovacuum_time, srcstats); + RELFSTAT_ACC(total_analyze_time, srcstats); + RELFSTAT_ACC(total_autoanalyze_time, srcstats); +#undef RELFSTAT_ACC + + /* Replace those statistics */ +#define RELFSTAT_REP(fld, stats_to_rep) \ + (dstshstats->stats.fld = stats_to_rep->fld) + RELFSTAT_REP(lastscan, srcstats); + RELFSTAT_REP(last_vacuum_time, srcstats); + RELFSTAT_REP(last_autovacuum_time, srcstats); + RELFSTAT_REP(last_analyze_time, srcstats); + RELFSTAT_REP(last_autoanalyze_time, srcstats); +#undef RELFSTAT_REP + } pgstat_unlock_entry(dst_ref); } @@ -136,6 +194,7 @@ void pgstat_assoc_relation(Relation rel) { RelFileLocator locator; + PgStat_TableStatus *pgstat_info; Assert(rel->pgstat_enabled); Assert(rel->pgstat_info == NULL); @@ -164,14 +223,54 @@ pgstat_assoc_relation(Relation rel) locator.relNumber = rel->rd_id; } + /* + * If this relation was rewritten during the current transaction we may be + * reopening it with its new RelFileLocator. In that case, continue using + * the stats entry associated with the old locator rather than creating a + * new one. This ensures all stats from before and after the rewrite are + * tracked in a single entry which will be properly copied to the new + * locator at transaction commit. + */ + if (pending_rewrites != NULL) + { + PgStat_PendingRewrite *rewrite; + + for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next) + { + if (locator.dbOid == rewrite->new_locator.dbOid && + locator.spcOid == rewrite->new_locator.spcOid && + locator.relNumber == rewrite->new_locator.relNumber) + { + pgstat_info = pgstat_prep_relation_pending(rewrite->old_locator); + goto found_entry; + } + } + } + /* Else find or make the PgStat_TableStatus entry, and update link */ - rel->pgstat_info = pgstat_prep_relation_pending(locator); + pgstat_info = pgstat_prep_relation_pending(locator); + +found_entry: + rel->pgstat_info = pgstat_info; + + /* + * For relations stats, we key by physical file location, not by relation + * OID. This means during operations like ALTER TYPE it's possible that + * the relation OID changes but the relfilenode stays the same (no actual + * rewrite needed). Unlink the old relation first. + */ + if (pgstat_info->relation != NULL && + pgstat_info->relation != rel) + { + pgstat_info->relation->pgstat_info = NULL; + pgstat_info->relation = NULL; + } /* don't allow link a stats to multiple relcache entries */ - Assert(rel->pgstat_info->relation == NULL); + Assert(pgstat_info->relation == NULL); /* mark this relation as the owner */ - rel->pgstat_info->relation = rel; + pgstat_info->relation = rel; } /* @@ -214,14 +313,37 @@ pgstat_drop_relation(Relation rel) { int nest_level = GetCurrentTransactionNestLevel(); PgStat_TableStatus *pgstat_info; + bool skip_transactional_drop = false; /* don't track stats for relations without storage */ if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind)) return; - pgstat_drop_transactional(PGSTAT_KIND_RELATION, - rel->rd_locator.dbOid, - RelFileLocatorToPgStatObjid(rel->rd_locator)); + /* Check if this drop is part of a pending rewrite */ + if (pending_rewrites != NULL) + { + PgStat_PendingRewrite *rewrite; + + for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next) + { + if (rel->rd_locator.dbOid == rewrite->old_locator.dbOid && + rel->rd_locator.spcOid == rewrite->old_locator.spcOid && + rel->rd_locator.relNumber == rewrite->old_locator.relNumber) + { + skip_transactional_drop = true; + break; + } + } + } + + /* + * If it is part of a rewrite, drop its stats later, for example in + * AtEOXact_PgStat_Relations(), so skip it here. + */ + if (!skip_transactional_drop) + pgstat_drop_transactional(PGSTAT_KIND_RELATION, + rel->rd_locator.dbOid, + RelFileLocatorToPgStatObjid(rel->rd_locator)); if (!pgstat_should_count_relation(rel)) return; @@ -666,6 +788,48 @@ AtEOXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit) } tabstat->trans = NULL; } + + /* preserve the stats in case of rewrite */ + if (isCommit && pending_rewrites != NULL) + { + PgStat_PendingRewrite *rewrite; + PgStat_PendingRewrite *prev = NULL; + PgStat_PendingRewrite *current = pending_rewrites; + PgStat_PendingRewrite *next; + + /* reverse the rewrites list to process in chronological order */ + while (current != NULL) + { + next = current->next; + current->next = prev; + prev = current; + current = next; + } + + /* now process rewrites in chronological order */ + for (rewrite = prev; rewrite != NULL; rewrite = rewrite->next) + { + PgStat_EntryRef *old_entry_ref; + + old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, + rewrite->old_locator.dbOid, + RelFileLocatorToPgStatObjid(rewrite->old_locator)); + + if (old_entry_ref && old_entry_ref->pending) + pgstat_relation_flush_cb(old_entry_ref, false); + + pgstat_copy_relation_stats(rewrite->new_locator, + rewrite->old_locator, true); + + /* drop old locator's stats */ + if (!pgstat_drop_entry(PGSTAT_KIND_RELATION, + rewrite->old_locator.dbOid, + RelFileLocatorToPgStatObjid(rewrite->old_locator))) + pgstat_request_entry_refs_gc(); + } + } + + pending_rewrites = NULL; } /* @@ -681,6 +845,30 @@ AtEOSubXact_PgStat_Relations(PgStat_SubXactStatus *xact_state, bool isCommit, in PgStat_TableXactStatus *trans; PgStat_TableXactStatus *next_trans; + /* + * If we don't commit then remove the associated rewrites if any, to keep + * the rewrite chain in sync with what will be eventually committed. + */ + if (!isCommit) + { + PgStat_PendingRewrite **rewrite_ptr = &pending_rewrites; + + while (*rewrite_ptr != NULL) + { + if ((*rewrite_ptr)->nest_level >= nestDepth) + { + PgStat_PendingRewrite *to_remove = *rewrite_ptr; + + *rewrite_ptr = (*rewrite_ptr)->next; + pfree(to_remove); + } + else + { + rewrite_ptr = &((*rewrite_ptr)->next); + } + } + } + for (trans = xact_state->first; trans != NULL; trans = next_trans) { PgStat_TableStatus *tabstat; @@ -760,11 +948,19 @@ void AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state) { PgStat_TableXactStatus *trans; + PgStat_PendingRewrite *rewrite; + /* + * For each tabstat, find its matching rewrite and remove it from the + * pending rewrites list. This way, after processing all tabstats, pending + * rewrites will only contain rewrite only transactions. + */ for (trans = xact_state->first; trans != NULL; trans = trans->next) { PgStat_TableStatus *tabstat PG_USED_FOR_ASSERTS_ONLY; TwoPhasePgStatRecord record; + PgStat_PendingRewrite **rewrite_ptr; + bool found_rewrite = false; Assert(trans->nest_level == 1); Assert(trans->upper == NULL); @@ -784,10 +980,83 @@ AtPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state) record.locator = tabstat->locator; record.truncdropped = trans->truncdropped; + record.rewrite_nest_level = 0; + + /* + * Look for a matching rewrite and remove it from pending rewrites. We + * check three possible matches: + * + * The new_locator when stats have been added after the rewrite. The + * old_locator when stats have been added before the rewrite but not + * after. The original_locator when this tabstat is part of a rewrite + * chain. + */ + rewrite_ptr = &pending_rewrites; + while (*rewrite_ptr != NULL) + { + rewrite = *rewrite_ptr; + + if ((record.locator.dbOid == rewrite->new_locator.dbOid && + record.locator.spcOid == rewrite->new_locator.spcOid && + record.locator.relNumber == rewrite->new_locator.relNumber) || + (tabstat->locator.dbOid == rewrite->old_locator.dbOid && + tabstat->locator.spcOid == rewrite->old_locator.spcOid && + tabstat->locator.relNumber == rewrite->old_locator.relNumber) || + (tabstat->locator.dbOid == rewrite->original_locator.dbOid && + tabstat->locator.spcOid == rewrite->original_locator.spcOid && + tabstat->locator.relNumber == rewrite->original_locator.relNumber)) + { + /* + * Found matching rewrite. Record the rewrite information and + * remove this rewrite from the list since it's now handled. + */ + record.rewrite_old_locator = rewrite->original_locator; + record.rewrite_nest_level = rewrite->nest_level; + record.locator = rewrite->new_locator; + found_rewrite = true; + + /* Remove from pending_rewrites list */ + *rewrite_ptr = rewrite->next; + pfree(rewrite); + break; + } + else + { + /* Move to next rewrite in the list */ + rewrite_ptr = &(rewrite->next); + } + } + + /* If no rewrite found, clear the rewrite fields */ + if (!found_rewrite) + { + memset(&record.rewrite_old_locator, 0, sizeof(RelFileLocator)); + } + + RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0, + &record, sizeof(TwoPhasePgStatRecord)); + } + + /* + * Now process any rewrites still pending. These are rewrite only + * transactions. We need to preserve their stats even though there's no + * tabstat entry for them. + */ + for (rewrite = pending_rewrites; rewrite != NULL; rewrite = rewrite->next) + { + TwoPhasePgStatRecord record; + + memset(&record, 0, sizeof(TwoPhasePgStatRecord)); + record.locator = rewrite->new_locator; + record.rewrite_old_locator = rewrite->original_locator; + record.rewrite_nest_level = rewrite->nest_level; + record.truncdropped = false; RegisterTwoPhaseRecord(TWOPHASE_RM_PGSTAT_ID, 0, &record, sizeof(TwoPhasePgStatRecord)); } + + pending_rewrites = NULL; } /* @@ -810,6 +1079,8 @@ PostPrepare_PgStat_Relations(PgStat_SubXactStatus *xact_state) tabstat = trans->parent; tabstat->trans = NULL; } + + pending_rewrites = NULL; } /* @@ -845,6 +1116,29 @@ pgstat_twophase_postcommit(FullTransactionId fxid, uint16 info, pgstat_info->counts.changed_tuples += rec->tuples_inserted + rec->tuples_updated + rec->tuples_deleted; + + if (rec->rewrite_nest_level > 0) + { + PgStat_EntryRef *old_entry_ref; + + /* Flush any pending stats for old locator first */ + old_entry_ref = pgstat_fetch_pending_entry(PGSTAT_KIND_RELATION, + rec->rewrite_old_locator.dbOid, + RelFileLocatorToPgStatObjid(rec->rewrite_old_locator)); + + if (old_entry_ref && old_entry_ref->pending) + pgstat_relation_flush_cb(old_entry_ref, false); + + /* Copy stats from old to new locator */ + pgstat_copy_relation_stats(rec->locator, rec->rewrite_old_locator, + true); + + /* Drop old locator's stats */ + if (!pgstat_drop_entry(PGSTAT_KIND_RELATION, + rec->rewrite_old_locator.dbOid, + RelFileLocatorToPgStatObjid(rec->rewrite_old_locator))) + pgstat_request_entry_refs_gc(); + } } /* @@ -859,9 +1153,26 @@ pgstat_twophase_postabort(FullTransactionId fxid, uint16 info, { TwoPhasePgStatRecord *rec = (TwoPhasePgStatRecord *) recdata; PgStat_TableStatus *pgstat_info; + RelFileLocator target_locator; + + /* + * For aborted transactions with rewrites (like TRUNCATE), we need to + * restore stats to the old locator, not the new one. The new locator + * should be dropped since the rewrite is being rolled back. + */ + if (rec->rewrite_nest_level > 0) + { + /* Use the old locator */ + target_locator = rec->rewrite_old_locator; + } + else + { + /* No rewrite, use the original locator */ + target_locator = rec->locator; + } /* Find or create a tabstat entry for the target locator */ - pgstat_info = pgstat_prep_relation_pending(rec->locator); + pgstat_info = pgstat_prep_relation_pending(target_locator); /* Same math as in AtEOXact_PgStat, abort case */ if (rec->truncdropped) @@ -916,7 +1227,17 @@ pgstat_relation_flush_cb(PgStat_EntryRef *entry_ref, bool nowait) tabentry->numscans += lstats->counts.numscans; if (lstats->counts.numscans) { - TimestampTz t = GetCurrentTransactionStopTimestamp(); + TimestampTz t; + + /* + * Checking the transaction state due to the flush call in + * pgstat_twophase_postcommit() that would break the assertion on the + * state in GetCurrentTransactionStopTimestamp(). + */ + if (!IsTransactionState()) + t = GetCurrentTransactionStopTimestamp(); + else + t = GetCurrentTimestamp(); if (t > tabentry->lastscan) tabentry->lastscan = t; @@ -1167,3 +1488,45 @@ pgstat_reloid_to_relfilelocator(Oid reloid, RelFileLocator *locator) ReleaseSysCache(tuple); return result; } + +/* + * Mark that a relation rewrite has occurred, preserving the original locator + * so stats can be copied at transaction commit. + */ +void +pgstat_mark_rewrite(RelFileLocator old_locator, RelFileLocator new_locator) +{ + PgStat_PendingRewrite *rewrite; + PgStat_PendingRewrite *existing; + RelFileLocator original_locator = old_locator; + + for (existing = pending_rewrites; existing != NULL; existing = existing->next) + { + if (old_locator.dbOid == existing->new_locator.dbOid && + old_locator.spcOid == existing->new_locator.spcOid && + old_locator.relNumber == existing->new_locator.relNumber) + { + original_locator = existing->original_locator; + break; + } + } + + /* Allocate in TopTransactionContext memory context */ + rewrite = MemoryContextAlloc(TopTransactionContext, + sizeof(PgStat_PendingRewrite)); + + rewrite->old_locator = old_locator; + rewrite->new_locator = new_locator; + rewrite->original_locator = original_locator; + rewrite->nest_level = GetCurrentTransactionNestLevel(); + + /* Add to the list */ + rewrite->next = pending_rewrites; + pending_rewrites = rewrite; +} + +void +pgstat_clear_rewrite(void) +{ + pending_rewrites = NULL; +} diff --git a/src/backend/utils/activity/pgstat_xact.c b/src/backend/utils/activity/pgstat_xact.c index bc9864bd8d9..f8cf3755ce2 100644 --- a/src/backend/utils/activity/pgstat_xact.c +++ b/src/backend/utils/activity/pgstat_xact.c @@ -55,6 +55,8 @@ AtEOXact_PgStat(bool isCommit, bool parallel) } pgStatXactStack = NULL; + pgstat_clear_rewrite(); + /* Make sure any stats snapshot is thrown away */ pgstat_clear_snapshot(); } @@ -360,8 +362,29 @@ create_drop_transactional_internal(PgStat_Kind kind, Oid dboid, uint64 objid, bo void pgstat_create_transactional(PgStat_Kind kind, Oid dboid, uint64 objid) { - if (pgstat_get_entry_ref(kind, dboid, objid, false, NULL)) + PgStat_EntryRef *entry_ref; + + entry_ref = pgstat_get_entry_ref(kind, dboid, objid, false, NULL); + + if (entry_ref) { + /* + * For relations stats, we key by physical file location, not by + * relation OID. This means during operations like ALTER TYPE where + * the relation OID changes but the relfilenode stays the same (no + * actual rewrite needed), we'll find an existing entry. + * + * This is expected behavior, we want to preserve stats across the + * catalog change. Simply reset and recreate the entry for the new + * relation OID without warning. + */ + if (kind == PGSTAT_KIND_RELATION) + { + pgstat_reset(kind, dboid, objid); + create_drop_transactional_internal(kind, dboid, objid, true); + return; + } + ereport(WARNING, errmsg("resetting existing statistics for kind %s, db=%u, oid=%" PRIu64, (pgstat_get_kind_info(kind))->name, dboid, diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 2d0cb7bcfd4..c98e5c51d63 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -85,6 +85,7 @@ #include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" +#include "utils/pgstat_internal.h" #include "utils/relmapper.h" #include "utils/resowner.h" #include "utils/snapmgr.h" @@ -3780,6 +3781,7 @@ RelationSetNewRelfilenumber(Relation relation, char persistence) MultiXactId minmulti = InvalidMultiXactId; TransactionId freezeXid = InvalidTransactionId; RelFileLocator newrlocator; + RelFileLocator oldrlocator = relation->rd_locator; if (!IsBinaryUpgrade) { @@ -3951,6 +3953,10 @@ RelationSetNewRelfilenumber(Relation relation, char persistence) table_close(pg_class, RowExclusiveLock); + /* Mark that a rewrite happened */ + if (RELKIND_HAS_STORAGE(relation->rd_rel->relkind)) + pgstat_mark_rewrite(oldrlocator, newrlocator); + /* * Make the pg_class row change or relation map change visible. This will * cause the relcache entry to get updated, too. diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 3102f86aa24..8750c025bbe 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -670,7 +670,7 @@ extern PgStat_FunctionCounts *find_funcstat_entry(Oid func_id); extern void pgstat_create_relation(Relation rel); extern void pgstat_drop_relation(Relation rel); -extern void pgstat_copy_relation_stats(Relation dst, Relation src); +extern void pgstat_copy_relation_stats(RelFileLocator dst, RelFileLocator src, bool increment); extern void pgstat_init_relation(Relation rel); extern void pgstat_assoc_relation(Relation rel); @@ -682,6 +682,9 @@ extern void pgstat_report_vacuum(Relation rel, PgStat_Counter livetuples, extern void pgstat_report_analyze(Relation rel, PgStat_Counter livetuples, PgStat_Counter deadtuples, bool resetcounter, TimestampTz starttime); +extern void pgstat_mark_rewrite(RelFileLocator old_locator, + RelFileLocator new_locator); +extern void pgstat_clear_rewrite(void); /* * If stats are enabled, but pending data hasn't been prepared yet, call diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 04845d5e680..3184e9c464b 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2269,6 +2269,7 @@ PgStat_KindInfo PgStat_LocalState PgStat_PendingDroppedStatsItem PgStat_PendingIO +PgStat_PendingRewrite PgStat_SLRUStats PgStat_ShmemControl PgStat_Snapshot -- 2.34.1 --xJsrX25x3tm7Tn81-- ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v6 04/14] bufmgr: Change BufferDesc.state to be a 64bit atomic @ 2025-11-06 00:22 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2025-11-06 00:22 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 88 ++++++---- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 158 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- src/test/modules/test_aio/test_aio.c | 12 +- 7 files changed, 192 insertions(+), 172 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 5400c56a965..28519ad2813 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * - 18 bits refcount * - 4 bits usage count @@ -39,6 +39,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -49,15 +52,21 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, "parts of buffer state space need to equal 32"); #define BUF_REFCOUNT_ONE 1 -#define BUF_REFCOUNT_MASK ((1U << BUF_REFCOUNT_BITS) - 1) -#define BUF_USAGECOUNT_MASK (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) -#define BUF_USAGECOUNT_ONE (1U << BUF_REFCOUNT_BITS) +#define BUF_REFCOUNT_MASK \ + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) +#define BUF_USAGECOUNT_MASK \ + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) +#define BUF_USAGECOUNT_ONE \ + (UINT64CONST(1) << BUF_REFCOUNT_BITS) #define BUF_USAGECOUNT_SHIFT BUF_REFCOUNT_BITS -#define BUF_FLAG_MASK (((1U << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) +#define BUF_FLAG_MASK \ + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) /* Get refcount and usagecount from buffer state */ -#define BUF_STATE_GET_REFCOUNT(state) ((state) & BUF_REFCOUNT_MASK) -#define BUF_STATE_GET_USAGECOUNT(state) (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) +#define BUF_STATE_GET_REFCOUNT(state) \ + ((uint32)((state) & BUF_REFCOUNT_MASK)) +#define BUF_STATE_GET_USAGECOUNT(state) \ + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -65,17 +74,28 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, * Note: BM_TAG_VALID essentially means that there is a buffer hashtable * entry associated with the buffer's tag. */ -#define BM_LOCKED (1U << 22) /* buffer header is locked */ -#define BM_DIRTY (1U << 23) /* data needs writing */ -#define BM_VALID (1U << 24) /* data is valid */ -#define BM_TAG_VALID (1U << 25) /* tag is assigned */ -#define BM_IO_IN_PROGRESS (1U << 26) /* read or write in progress */ -#define BM_IO_ERROR (1U << 27) /* previous I/O failed */ -#define BM_JUST_DIRTIED (1U << 28) /* dirtied since write started */ -#define BM_PIN_COUNT_WAITER (1U << 29) /* have waiter for sole pin */ -#define BM_CHECKPOINT_NEEDED (1U << 30) /* must write for checkpoint */ -#define BM_PERMANENT (1U << 31) /* permanent buffer (not unlogged, - * or init fork) */ + +/* buffer header is locked */ +#define BM_LOCKED (UINT64CONST(1) << 22) +/* data needs writing */ +#define BM_DIRTY (UINT64CONST(1) << 23) +/* data is valid */ +#define BM_VALID (UINT64CONST(1) << 24) +/* tag is assigned */ +#define BM_TAG_VALID (UINT64CONST(1) << 25) +/* read or write in progress */ +#define BM_IO_IN_PROGRESS (UINT64CONST(1) << 26) +/* previous I/O failed */ +#define BM_IO_ERROR (UINT64CONST(1) << 27) +/* dirtied since write started */ +#define BM_JUST_DIRTIED (UINT64CONST(1) << 28) +/* have waiter for sole pin */ +#define BM_PIN_COUNT_WAITER (UINT64CONST(1) << 29) +/* must write for checkpoint */ +#define BM_CHECKPOINT_NEEDED (UINT64CONST(1) << 30) +/* permanent buffer (not unlogged, or init fork) */ +#define BM_PERMANENT (UINT64CONST(1) << 31) + /* * The maximum allowed value of usage_count represents a tradeoff between * accuracy and speed of the clock-sweep buffer management algorithm. A @@ -86,7 +106,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -251,8 +271,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -280,7 +300,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -386,7 +406,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -397,9 +417,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -410,14 +430,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -428,7 +448,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -436,7 +456,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -496,14 +516,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -539,7 +559,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 6fd3a6bbac5..25f71191ec3 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index b682878b1fb..e33fa0cbfec 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -686,7 +686,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -699,7 +699,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1292,8 +1292,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1519,10 +1519,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -1989,8 +1989,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2157,7 +2157,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2248,7 +2248,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2308,10 +2308,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2321,7 +2321,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2454,7 +2454,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2745,13 +2745,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -2927,7 +2927,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -2943,8 +2943,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -2964,7 +2964,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -2975,7 +2975,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3079,10 +3079,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3116,7 +3116,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3143,7 +3143,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->refcount > 0); ref->refcount++; @@ -3178,7 +3178,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3190,7 +3190,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3220,7 +3220,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3267,7 +3267,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->refcount--; if (ref->refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3285,7 +3285,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3342,7 +3342,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3352,7 +3352,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3384,7 +3384,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3551,7 +3551,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -3921,7 +3921,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4169,7 +4169,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4186,9 +4186,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufhdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4288,7 +4288,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4486,7 +4486,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -4949,11 +4949,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -4989,7 +4989,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5061,7 +5061,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5310,7 +5310,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5458,13 +5458,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5476,7 +5476,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5576,8 +5576,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5708,8 +5708,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5857,7 +5857,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -5915,7 +5915,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -5971,7 +5971,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6045,7 +6045,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6101,11 +6101,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6166,7 +6166,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6260,11 +6260,11 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { SpinDelayStatus delayStatus; - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6273,7 +6273,7 @@ LockBufHdr(BufferDesc *desc) while (true) { /* set BM_LOCKED flag */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); /* if it wasn't set before we're OK */ if (!(old_buf_state & BM_LOCKED)) break; @@ -6290,20 +6290,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6591,12 +6591,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6690,12 +6690,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6742,7 +6742,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6809,7 +6809,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -6827,7 +6827,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -6865,7 +6865,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7051,13 +7051,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 28d952b3534..1d4f19a9afd 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 15aac7d1c9f..a41a5facd3a 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index c29b784dfa1..32bd8aa784a 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -192,7 +192,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -559,7 +559,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -570,7 +570,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -620,7 +620,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index d7eadeab256..488d98e7e66 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --mhhc7c45w3dmihtp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0005-Rename-BUFFERPIN-wait-event-class-to-BUFFER.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v7 11/15] bufmgr: Change BufferDesc.state to be a 64bit atomic @ 2025-12-02 23:46 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2025-12-02 23:46 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 88 +++++---- src/include/storage/procnumber.h | 14 +- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 170 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- src/test/modules/test_aio/test_aio.c | 12 +- 8 files changed, 205 insertions(+), 185 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 5400c56a965..28519ad2813 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * - 18 bits refcount * - 4 bits usage count @@ -39,6 +39,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -49,15 +52,21 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, "parts of buffer state space need to equal 32"); #define BUF_REFCOUNT_ONE 1 -#define BUF_REFCOUNT_MASK ((1U << BUF_REFCOUNT_BITS) - 1) -#define BUF_USAGECOUNT_MASK (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) -#define BUF_USAGECOUNT_ONE (1U << BUF_REFCOUNT_BITS) +#define BUF_REFCOUNT_MASK \ + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) +#define BUF_USAGECOUNT_MASK \ + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) +#define BUF_USAGECOUNT_ONE \ + (UINT64CONST(1) << BUF_REFCOUNT_BITS) #define BUF_USAGECOUNT_SHIFT BUF_REFCOUNT_BITS -#define BUF_FLAG_MASK (((1U << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) +#define BUF_FLAG_MASK \ + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) /* Get refcount and usagecount from buffer state */ -#define BUF_STATE_GET_REFCOUNT(state) ((state) & BUF_REFCOUNT_MASK) -#define BUF_STATE_GET_USAGECOUNT(state) (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) +#define BUF_STATE_GET_REFCOUNT(state) \ + ((uint32)((state) & BUF_REFCOUNT_MASK)) +#define BUF_STATE_GET_USAGECOUNT(state) \ + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -65,17 +74,28 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, * Note: BM_TAG_VALID essentially means that there is a buffer hashtable * entry associated with the buffer's tag. */ -#define BM_LOCKED (1U << 22) /* buffer header is locked */ -#define BM_DIRTY (1U << 23) /* data needs writing */ -#define BM_VALID (1U << 24) /* data is valid */ -#define BM_TAG_VALID (1U << 25) /* tag is assigned */ -#define BM_IO_IN_PROGRESS (1U << 26) /* read or write in progress */ -#define BM_IO_ERROR (1U << 27) /* previous I/O failed */ -#define BM_JUST_DIRTIED (1U << 28) /* dirtied since write started */ -#define BM_PIN_COUNT_WAITER (1U << 29) /* have waiter for sole pin */ -#define BM_CHECKPOINT_NEEDED (1U << 30) /* must write for checkpoint */ -#define BM_PERMANENT (1U << 31) /* permanent buffer (not unlogged, - * or init fork) */ + +/* buffer header is locked */ +#define BM_LOCKED (UINT64CONST(1) << 22) +/* data needs writing */ +#define BM_DIRTY (UINT64CONST(1) << 23) +/* data is valid */ +#define BM_VALID (UINT64CONST(1) << 24) +/* tag is assigned */ +#define BM_TAG_VALID (UINT64CONST(1) << 25) +/* read or write in progress */ +#define BM_IO_IN_PROGRESS (UINT64CONST(1) << 26) +/* previous I/O failed */ +#define BM_IO_ERROR (UINT64CONST(1) << 27) +/* dirtied since write started */ +#define BM_JUST_DIRTIED (UINT64CONST(1) << 28) +/* have waiter for sole pin */ +#define BM_PIN_COUNT_WAITER (UINT64CONST(1) << 29) +/* must write for checkpoint */ +#define BM_CHECKPOINT_NEEDED (UINT64CONST(1) << 30) +/* permanent buffer (not unlogged, or init fork) */ +#define BM_PERMANENT (UINT64CONST(1) << 31) + /* * The maximum allowed value of usage_count represents a tradeoff between * accuracy and speed of the clock-sweep buffer management algorithm. A @@ -86,7 +106,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -251,8 +271,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -280,7 +300,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -386,7 +406,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -397,9 +417,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -410,14 +430,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -428,7 +448,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -436,7 +456,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -496,14 +516,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -539,7 +559,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/include/storage/procnumber.h b/src/include/storage/procnumber.h index 2ddaaf0c646..6baac7c77f1 100644 --- a/src/include/storage/procnumber.h +++ b/src/include/storage/procnumber.h @@ -27,13 +27,13 @@ typedef int ProcNumber; /* * Note: MAX_BACKENDS_BITS is 18 as that is the space available for buffer - * refcounts in buf_internals.h. This limitation could be lifted by using a - * 64bit state; but it's unlikely to be worthwhile as 2^18-1 backends exceed - * currently realistic configurations. Even if that limitation were removed, - * we still could not a) exceed 2^23-1 because inval.c stores the ProcNumber - * as a 3-byte signed integer, b) INT_MAX/4 because some places compute - * 4*MaxBackends without any overflow check. We check that the configured - * number of backends does not exceed MAX_BACKENDS in InitializeMaxBackends(). + * refcounts in buf_internals.h. This limitation could be lifted, but it's + * unlikely to be worthwhile as 2^18-1 backends exceed currently realistic + * configurations. Even if that limitation were removed, we still could not a) + * exceed 2^23-1 because inval.c stores the ProcNumber as a 3-byte signed + * integer, b) INT_MAX/4 because some places compute 4*MaxBackends without any + * overflow check. We check that the configured number of backends does not + * exceed MAX_BACKENDS in InitializeMaxBackends(). */ #define MAX_BACKENDS_BITS 18 #define MAX_BACKENDS ((1U << MAX_BACKENDS_BITS)-1) diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 6fd3a6bbac5..25f71191ec3 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index be32bd596f6..d0b8f8d20eb 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -775,7 +775,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -788,7 +788,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1381,8 +1381,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1608,10 +1608,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -2078,8 +2078,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2246,7 +2246,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2337,7 +2337,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2397,10 +2397,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2410,7 +2410,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2543,7 +2543,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2834,13 +2834,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -3016,7 +3016,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -3032,8 +3032,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -3053,7 +3053,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -3064,7 +3064,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3168,10 +3168,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3205,7 +3205,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3232,7 +3232,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->data.refcount > 0); ref->data.refcount++; @@ -3267,7 +3267,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3279,7 +3279,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3309,7 +3309,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3356,7 +3356,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->data.refcount--; if (ref->data.refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3374,7 +3374,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3431,7 +3431,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3441,7 +3441,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3473,7 +3473,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3640,7 +3640,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -4010,7 +4010,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4259,7 +4259,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4276,9 +4276,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufhdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4378,7 +4378,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4576,7 +4576,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -5039,11 +5039,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -5079,7 +5079,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5151,7 +5151,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5400,7 +5400,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5548,13 +5548,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5566,7 +5566,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5666,8 +5666,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5798,8 +5798,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5947,7 +5947,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -6005,7 +6005,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -6061,7 +6061,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6135,7 +6135,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6191,11 +6191,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6256,7 +6256,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6350,10 +6350,10 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6362,7 +6362,7 @@ LockBufHdr(BufferDesc *desc) * infrastructure. The work necessary for that shows up in profiles and is * rarely necessary. */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); if (unlikely(old_buf_state & BM_LOCKED)) { @@ -6373,7 +6373,7 @@ LockBufHdr(BufferDesc *desc) while (true) { /* set BM_LOCKED flag */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); /* if it wasn't set before we're OK */ if (!(old_buf_state & BM_LOCKED)) break; @@ -6392,20 +6392,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6693,12 +6693,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6792,12 +6792,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6844,7 +6844,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6886,12 +6886,12 @@ static bool MarkDirtyUnpinnedBufferInternal(Buffer buf, BufferDesc *desc, bool *buffer_already_dirty) { - uint32 buf_state; + uint64 buf_state; bool result = false; *buffer_already_dirty = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6989,7 +6989,7 @@ MarkDirtyRelUnpinnedBuffers(Relation rel, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); @@ -7043,12 +7043,12 @@ MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -7099,7 +7099,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -7117,7 +7117,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -7155,7 +7155,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7341,13 +7341,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 28d952b3534..1d4f19a9afd 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 15aac7d1c9f..a41a5facd3a 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 702307a49e2..294caf7a1eb 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -199,7 +199,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -615,7 +615,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -626,7 +626,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -676,7 +676,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index d7eadeab256..488d98e7e66 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --kad4mm4awdo7v2ki Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v7-0012-bufmgr-Implement-buffer-content-locks-independent.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v8 06/10] bufmgr: Change BufferDesc.state to be a 64bit atomic @ 2025-12-18 23:28 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2025-12-18 23:28 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 88 +++++---- src/include/storage/procnumber.h | 14 +- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 170 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- src/test/modules/test_aio/test_aio.c | 12 +- 8 files changed, 205 insertions(+), 185 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 5400c56a965..28519ad2813 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * - 18 bits refcount * - 4 bits usage count @@ -39,6 +39,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -49,15 +52,21 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, "parts of buffer state space need to equal 32"); #define BUF_REFCOUNT_ONE 1 -#define BUF_REFCOUNT_MASK ((1U << BUF_REFCOUNT_BITS) - 1) -#define BUF_USAGECOUNT_MASK (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) -#define BUF_USAGECOUNT_ONE (1U << BUF_REFCOUNT_BITS) +#define BUF_REFCOUNT_MASK \ + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) +#define BUF_USAGECOUNT_MASK \ + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_REFCOUNT_BITS)) +#define BUF_USAGECOUNT_ONE \ + (UINT64CONST(1) << BUF_REFCOUNT_BITS) #define BUF_USAGECOUNT_SHIFT BUF_REFCOUNT_BITS -#define BUF_FLAG_MASK (((1U << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) +#define BUF_FLAG_MASK \ + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS)) /* Get refcount and usagecount from buffer state */ -#define BUF_STATE_GET_REFCOUNT(state) ((state) & BUF_REFCOUNT_MASK) -#define BUF_STATE_GET_USAGECOUNT(state) (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) +#define BUF_STATE_GET_REFCOUNT(state) \ + ((uint32)((state) & BUF_REFCOUNT_MASK)) +#define BUF_STATE_GET_USAGECOUNT(state) \ + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -65,17 +74,28 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, * Note: BM_TAG_VALID essentially means that there is a buffer hashtable * entry associated with the buffer's tag. */ -#define BM_LOCKED (1U << 22) /* buffer header is locked */ -#define BM_DIRTY (1U << 23) /* data needs writing */ -#define BM_VALID (1U << 24) /* data is valid */ -#define BM_TAG_VALID (1U << 25) /* tag is assigned */ -#define BM_IO_IN_PROGRESS (1U << 26) /* read or write in progress */ -#define BM_IO_ERROR (1U << 27) /* previous I/O failed */ -#define BM_JUST_DIRTIED (1U << 28) /* dirtied since write started */ -#define BM_PIN_COUNT_WAITER (1U << 29) /* have waiter for sole pin */ -#define BM_CHECKPOINT_NEEDED (1U << 30) /* must write for checkpoint */ -#define BM_PERMANENT (1U << 31) /* permanent buffer (not unlogged, - * or init fork) */ + +/* buffer header is locked */ +#define BM_LOCKED (UINT64CONST(1) << 22) +/* data needs writing */ +#define BM_DIRTY (UINT64CONST(1) << 23) +/* data is valid */ +#define BM_VALID (UINT64CONST(1) << 24) +/* tag is assigned */ +#define BM_TAG_VALID (UINT64CONST(1) << 25) +/* read or write in progress */ +#define BM_IO_IN_PROGRESS (UINT64CONST(1) << 26) +/* previous I/O failed */ +#define BM_IO_ERROR (UINT64CONST(1) << 27) +/* dirtied since write started */ +#define BM_JUST_DIRTIED (UINT64CONST(1) << 28) +/* have waiter for sole pin */ +#define BM_PIN_COUNT_WAITER (UINT64CONST(1) << 29) +/* must write for checkpoint */ +#define BM_CHECKPOINT_NEEDED (UINT64CONST(1) << 30) +/* permanent buffer (not unlogged, or init fork) */ +#define BM_PERMANENT (UINT64CONST(1) << 31) + /* * The maximum allowed value of usage_count represents a tradeoff between * accuracy and speed of the clock-sweep buffer management algorithm. A @@ -86,7 +106,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -251,8 +271,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -280,7 +300,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -386,7 +406,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -397,9 +417,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -410,14 +430,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -428,7 +448,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -436,7 +456,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -496,14 +516,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -539,7 +559,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/include/storage/procnumber.h b/src/include/storage/procnumber.h index 2ddaaf0c646..6baac7c77f1 100644 --- a/src/include/storage/procnumber.h +++ b/src/include/storage/procnumber.h @@ -27,13 +27,13 @@ typedef int ProcNumber; /* * Note: MAX_BACKENDS_BITS is 18 as that is the space available for buffer - * refcounts in buf_internals.h. This limitation could be lifted by using a - * 64bit state; but it's unlikely to be worthwhile as 2^18-1 backends exceed - * currently realistic configurations. Even if that limitation were removed, - * we still could not a) exceed 2^23-1 because inval.c stores the ProcNumber - * as a 3-byte signed integer, b) INT_MAX/4 because some places compute - * 4*MaxBackends without any overflow check. We check that the configured - * number of backends does not exceed MAX_BACKENDS in InitializeMaxBackends(). + * refcounts in buf_internals.h. This limitation could be lifted, but it's + * unlikely to be worthwhile as 2^18-1 backends exceed currently realistic + * configurations. Even if that limitation were removed, we still could not a) + * exceed 2^23-1 because inval.c stores the ProcNumber as a 3-byte signed + * integer, b) INT_MAX/4 because some places compute 4*MaxBackends without any + * overflow check. We check that the configured number of backends does not + * exceed MAX_BACKENDS in InitializeMaxBackends(). */ #define MAX_BACKENDS_BITS 18 #define MAX_BACKENDS ((1U << MAX_BACKENDS_BITS)-1) diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 6fd3a6bbac5..25f71191ec3 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index eb55102b0d7..03d99d294d5 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -780,7 +780,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -793,7 +793,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1386,8 +1386,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1613,10 +1613,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -2083,8 +2083,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2251,7 +2251,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2342,7 +2342,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2402,10 +2402,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2415,7 +2415,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2548,7 +2548,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2839,13 +2839,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -3021,7 +3021,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -3037,8 +3037,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -3058,7 +3058,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -3069,7 +3069,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3173,10 +3173,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3210,7 +3210,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3237,7 +3237,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->data.refcount > 0); ref->data.refcount++; @@ -3272,7 +3272,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3284,7 +3284,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3314,7 +3314,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3361,7 +3361,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->data.refcount--; if (ref->data.refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3379,7 +3379,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3436,7 +3436,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3446,7 +3446,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3478,7 +3478,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3645,7 +3645,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -4015,7 +4015,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4264,7 +4264,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4281,9 +4281,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufhdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4383,7 +4383,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4581,7 +4581,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -5044,11 +5044,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -5084,7 +5084,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5156,7 +5156,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5405,7 +5405,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5553,13 +5553,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5571,7 +5571,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5671,8 +5671,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5803,8 +5803,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5952,7 +5952,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -6010,7 +6010,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -6066,7 +6066,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6140,7 +6140,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6196,11 +6196,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6261,7 +6261,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6355,10 +6355,10 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6369,7 +6369,7 @@ LockBufHdr(BufferDesc *desc) * the spin-delay infrastructure. The work necessary for that shows up * in profiles and is rarely necessary. */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); if (likely(!(old_buf_state & BM_LOCKED))) break; /* got lock */ @@ -6382,7 +6382,7 @@ LockBufHdr(BufferDesc *desc) while (old_buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - old_buf_state = pg_atomic_read_u32(&desc->state); + old_buf_state = pg_atomic_read_u64(&desc->state); } finish_spin_delay(&delayStatus); } @@ -6403,20 +6403,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6704,12 +6704,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6803,12 +6803,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6855,7 +6855,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6897,12 +6897,12 @@ static bool MarkDirtyUnpinnedBufferInternal(Buffer buf, BufferDesc *desc, bool *buffer_already_dirty) { - uint32 buf_state; + uint64 buf_state; bool result = false; *buffer_already_dirty = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -7000,7 +7000,7 @@ MarkDirtyRelUnpinnedBuffers(Relation rel, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); @@ -7054,12 +7054,12 @@ MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -7110,7 +7110,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -7128,7 +7128,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -7166,7 +7166,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7352,13 +7352,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 28d952b3534..1d4f19a9afd 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 15aac7d1c9f..a41a5facd3a 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 0c58e4b265c..529803346ce 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -199,7 +199,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -615,7 +615,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -626,7 +626,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -676,7 +676,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index d7eadeab256..488d98e7e66 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --lbbj3oq7cjvyhmbk Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v8-0007-bufmgr-Implement-buffer-content-locks-independent.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v9 06/10] bufmgr: Change BufferDesc.state to be a 64bit atomic @ 2026-01-07 22:26 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2026-01-07 22:26 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 51 +++--- src/include/storage/procnumber.h | 14 +- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 170 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- src/test/modules/test_aio/test_aio.c | 12 +- 8 files changed, 178 insertions(+), 175 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 2f607ea2ac5..a4d36e9ca01 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * State of the buffer itself (in order): * - 18 bits refcount @@ -40,6 +40,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -52,27 +55,27 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, /* refcount related definitions */ #define BUF_REFCOUNT_ONE 1 #define BUF_REFCOUNT_MASK \ - ((1U << BUF_REFCOUNT_BITS) - 1) + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) /* usage count related definitions */ #define BUF_USAGECOUNT_SHIFT \ BUF_REFCOUNT_BITS #define BUF_USAGECOUNT_MASK \ - (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) #define BUF_USAGECOUNT_ONE \ - (1U << BUF_REFCOUNT_BITS) + (UINT64CONST(1) << BUF_REFCOUNT_BITS) /* flags related definitions */ #define BUF_FLAG_SHIFT \ (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS) #define BUF_FLAG_MASK \ - (((1U << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) /* Get refcount and usagecount from buffer state */ #define BUF_STATE_GET_REFCOUNT(state) \ - ((state) & BUF_REFCOUNT_MASK) + ((uint32)((state) & BUF_REFCOUNT_MASK)) #define BUF_STATE_GET_USAGECOUNT(state) \ - (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -82,7 +85,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BUF_DEFINE_FLAG(flagno) \ - (1U << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + (flagno))) + (UINT64CONST(1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + (flagno))) /* buffer header is locked */ #define BM_LOCKED BUF_DEFINE_FLAG( 0) @@ -115,7 +118,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -280,8 +283,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -309,7 +312,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -415,7 +418,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -426,9 +429,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -439,14 +442,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -457,7 +460,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -465,7 +468,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -525,14 +528,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -568,7 +571,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/include/storage/procnumber.h b/src/include/storage/procnumber.h index 30c360ad350..bd9cb3891cc 100644 --- a/src/include/storage/procnumber.h +++ b/src/include/storage/procnumber.h @@ -27,13 +27,13 @@ typedef int ProcNumber; /* * Note: MAX_BACKENDS_BITS is 18 as that is the space available for buffer - * refcounts in buf_internals.h. This limitation could be lifted by using a - * 64bit state; but it's unlikely to be worthwhile as 2^18-1 backends exceed - * currently realistic configurations. Even if that limitation were removed, - * we still could not a) exceed 2^23-1 because inval.c stores the ProcNumber - * as a 3-byte signed integer, b) INT_MAX/4 because some places compute - * 4*MaxBackends without any overflow check. We check that the configured - * number of backends does not exceed MAX_BACKENDS in InitializeMaxBackends(). + * refcounts in buf_internals.h. This limitation could be lifted, but it's + * unlikely to be worthwhile as 2^18-1 backends exceed currently realistic + * configurations. Even if that limitation were removed, we still could not a) + * exceed 2^23-1 because inval.c stores the ProcNumber as a 3-byte signed + * integer, b) INT_MAX/4 because some places compute 4*MaxBackends without any + * overflow check. We check that the configured number of backends does not + * exceed MAX_BACKENDS in InitializeMaxBackends(). */ #define MAX_BACKENDS_BITS 18 #define MAX_BACKENDS ((1U << MAX_BACKENDS_BITS)-1) diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 9a312bcc7b3..7d894522526 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index a036c2aa275..b0de8e45d4d 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -780,7 +780,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -793,7 +793,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1386,8 +1386,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1613,10 +1613,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -2083,8 +2083,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2251,7 +2251,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2342,7 +2342,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2402,10 +2402,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2415,7 +2415,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2548,7 +2548,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2839,13 +2839,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -3021,7 +3021,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -3037,8 +3037,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -3058,7 +3058,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -3069,7 +3069,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3173,10 +3173,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3210,7 +3210,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3237,7 +3237,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->data.refcount > 0); ref->data.refcount++; @@ -3272,7 +3272,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3284,7 +3284,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3314,7 +3314,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3361,7 +3361,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->data.refcount--; if (ref->data.refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3379,7 +3379,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3436,7 +3436,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3446,7 +3446,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3478,7 +3478,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3645,7 +3645,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -4015,7 +4015,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4264,7 +4264,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4281,9 +4281,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufHdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4383,7 +4383,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4581,7 +4581,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -5044,11 +5044,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -5084,7 +5084,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5156,7 +5156,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5405,7 +5405,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5553,13 +5553,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5571,7 +5571,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5671,8 +5671,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5803,8 +5803,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5952,7 +5952,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -6010,7 +6010,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -6066,7 +6066,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6140,7 +6140,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6196,11 +6196,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6261,7 +6261,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6355,10 +6355,10 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6369,7 +6369,7 @@ LockBufHdr(BufferDesc *desc) * the spin-delay infrastructure. The work necessary for that shows up * in profiles and is rarely necessary. */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); if (likely(!(old_buf_state & BM_LOCKED))) break; /* got lock */ @@ -6382,7 +6382,7 @@ LockBufHdr(BufferDesc *desc) while (old_buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - old_buf_state = pg_atomic_read_u32(&desc->state); + old_buf_state = pg_atomic_read_u64(&desc->state); } finish_spin_delay(&delayStatus); } @@ -6403,20 +6403,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6704,12 +6704,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6803,12 +6803,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6855,7 +6855,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6897,12 +6897,12 @@ static bool MarkDirtyUnpinnedBufferInternal(Buffer buf, BufferDesc *desc, bool *buffer_already_dirty) { - uint32 buf_state; + uint64 buf_state; bool result = false; *buffer_already_dirty = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -7000,7 +7000,7 @@ MarkDirtyRelUnpinnedBuffers(Relation rel, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); @@ -7054,12 +7054,12 @@ MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -7110,7 +7110,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -7128,7 +7128,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -7166,7 +7166,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7352,13 +7352,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 9a93fb335fc..b7687836188 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index f6e2b1aa288..04a540379a2 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 0c58e4b265c..529803346ce 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -199,7 +199,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -615,7 +615,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -626,7 +626,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -676,7 +676,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index e046b08f3d5..b1aa8af9ec0 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --rkiyqpij3ajqn7ww Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v9-0007-bufmgr-Implement-buffer-content-locks-independent.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v10 3/8] bufmgr: Change BufferDesc.state to be a 64-bit atomic @ 2026-01-07 22:26 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2026-01-07 22:26 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff for more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 51 +++--- src/include/storage/procnumber.h | 14 +- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 170 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- src/test/modules/test_aio/test_aio.c | 12 +- 8 files changed, 178 insertions(+), 175 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 2f607ea2ac5..a4d36e9ca01 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * State of the buffer itself (in order): * - 18 bits refcount @@ -40,6 +40,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -52,27 +55,27 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, /* refcount related definitions */ #define BUF_REFCOUNT_ONE 1 #define BUF_REFCOUNT_MASK \ - ((1U << BUF_REFCOUNT_BITS) - 1) + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) /* usage count related definitions */ #define BUF_USAGECOUNT_SHIFT \ BUF_REFCOUNT_BITS #define BUF_USAGECOUNT_MASK \ - (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) #define BUF_USAGECOUNT_ONE \ - (1U << BUF_REFCOUNT_BITS) + (UINT64CONST(1) << BUF_REFCOUNT_BITS) /* flags related definitions */ #define BUF_FLAG_SHIFT \ (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS) #define BUF_FLAG_MASK \ - (((1U << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) /* Get refcount and usagecount from buffer state */ #define BUF_STATE_GET_REFCOUNT(state) \ - ((state) & BUF_REFCOUNT_MASK) + ((uint32)((state) & BUF_REFCOUNT_MASK)) #define BUF_STATE_GET_USAGECOUNT(state) \ - (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -82,7 +85,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BUF_DEFINE_FLAG(flagno) \ - (1U << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + (flagno))) + (UINT64CONST(1) << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + (flagno))) /* buffer header is locked */ #define BM_LOCKED BUF_DEFINE_FLAG( 0) @@ -115,7 +118,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -280,8 +283,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -309,7 +312,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -415,7 +418,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -426,9 +429,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -439,14 +442,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -457,7 +460,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -465,7 +468,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -525,14 +528,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -568,7 +571,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/include/storage/procnumber.h b/src/include/storage/procnumber.h index 30c360ad350..bd9cb3891cc 100644 --- a/src/include/storage/procnumber.h +++ b/src/include/storage/procnumber.h @@ -27,13 +27,13 @@ typedef int ProcNumber; /* * Note: MAX_BACKENDS_BITS is 18 as that is the space available for buffer - * refcounts in buf_internals.h. This limitation could be lifted by using a - * 64bit state; but it's unlikely to be worthwhile as 2^18-1 backends exceed - * currently realistic configurations. Even if that limitation were removed, - * we still could not a) exceed 2^23-1 because inval.c stores the ProcNumber - * as a 3-byte signed integer, b) INT_MAX/4 because some places compute - * 4*MaxBackends without any overflow check. We check that the configured - * number of backends does not exceed MAX_BACKENDS in InitializeMaxBackends(). + * refcounts in buf_internals.h. This limitation could be lifted, but it's + * unlikely to be worthwhile as 2^18-1 backends exceed currently realistic + * configurations. Even if that limitation were removed, we still could not a) + * exceed 2^23-1 because inval.c stores the ProcNumber as a 3-byte signed + * integer, b) INT_MAX/4 because some places compute 4*MaxBackends without any + * overflow check. We check that the configured number of backends does not + * exceed MAX_BACKENDS in InitializeMaxBackends(). */ #define MAX_BACKENDS_BITS 18 #define MAX_BACKENDS ((1U << MAX_BACKENDS_BITS)-1) diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 9a312bcc7b3..7d894522526 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index a036c2aa275..b0de8e45d4d 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -780,7 +780,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -793,7 +793,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1386,8 +1386,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1613,10 +1613,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -2083,8 +2083,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2251,7 +2251,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2342,7 +2342,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2402,10 +2402,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2415,7 +2415,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2548,7 +2548,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2839,13 +2839,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -3021,7 +3021,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -3037,8 +3037,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -3058,7 +3058,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -3069,7 +3069,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3173,10 +3173,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3210,7 +3210,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3237,7 +3237,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->data.refcount > 0); ref->data.refcount++; @@ -3272,7 +3272,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3284,7 +3284,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3314,7 +3314,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3361,7 +3361,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->data.refcount--; if (ref->data.refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3379,7 +3379,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3436,7 +3436,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3446,7 +3446,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3478,7 +3478,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3645,7 +3645,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -4015,7 +4015,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4264,7 +4264,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4281,9 +4281,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufHdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4383,7 +4383,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4581,7 +4581,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -5044,11 +5044,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -5084,7 +5084,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5156,7 +5156,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5405,7 +5405,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5553,13 +5553,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5571,7 +5571,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5671,8 +5671,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5803,8 +5803,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5952,7 +5952,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -6010,7 +6010,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -6066,7 +6066,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6140,7 +6140,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6196,11 +6196,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6261,7 +6261,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6355,10 +6355,10 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6369,7 +6369,7 @@ LockBufHdr(BufferDesc *desc) * the spin-delay infrastructure. The work necessary for that shows up * in profiles and is rarely necessary. */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); if (likely(!(old_buf_state & BM_LOCKED))) break; /* got lock */ @@ -6382,7 +6382,7 @@ LockBufHdr(BufferDesc *desc) while (old_buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - old_buf_state = pg_atomic_read_u32(&desc->state); + old_buf_state = pg_atomic_read_u64(&desc->state); } finish_spin_delay(&delayStatus); } @@ -6403,20 +6403,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6704,12 +6704,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6803,12 +6803,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6855,7 +6855,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6897,12 +6897,12 @@ static bool MarkDirtyUnpinnedBufferInternal(Buffer buf, BufferDesc *desc, bool *buffer_already_dirty) { - uint32 buf_state; + uint64 buf_state; bool result = false; *buffer_already_dirty = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -7000,7 +7000,7 @@ MarkDirtyRelUnpinnedBuffers(Relation rel, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); @@ -7054,12 +7054,12 @@ MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -7110,7 +7110,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -7128,7 +7128,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -7166,7 +7166,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7352,13 +7352,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 9a93fb335fc..b7687836188 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index f6e2b1aa288..04a540379a2 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 0c58e4b265c..529803346ce 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -199,7 +199,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -615,7 +615,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -626,7 +626,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -676,7 +676,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index e046b08f3d5..b1aa8af9ec0 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --jmawlk4t5yqwiemy Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v10-0004-bufmgr-Implement-buffer-content-locks-independen.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
* [PATCH v11 1/7] bufmgr: Change BufferDesc.state to be a 64-bit atomic @ 2026-01-14 01:10 Andres Freund <[email protected]> 0 siblings, 0 replies; 7+ messages in thread From: Andres Freund @ 2026-01-14 01:10 UTC (permalink / raw) This is motivated by wanting to merge buffer content locks into BufferDesc.state in a future commit, rather than having a separate lwlock (see commit c75ebc657ff for more details). As this change is rather mechanical, it seems to make sense to split it out into a separate commit, for easier review. Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/fvfmkr5kk4nyex56ejgxj3uzi63isfxovp2biecb4bspbjrze7@az2pljabhnff --- src/include/storage/buf_internals.h | 51 +++--- src/include/storage/procnumber.h | 14 +- src/backend/storage/buffer/buf_init.c | 2 +- src/backend/storage/buffer/bufmgr.c | 170 +++++++++--------- src/backend/storage/buffer/freelist.c | 24 +-- src/backend/storage/buffer/localbuf.c | 72 ++++---- contrib/pg_buffercache/pg_buffercache_pages.c | 8 +- contrib/pg_prewarm/autoprewarm.c | 2 +- src/test/modules/test_aio/test_aio.c | 12 +- 9 files changed, 179 insertions(+), 176 deletions(-) diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h index 2f607ea2ac5..e6e788224f5 100644 --- a/src/include/storage/buf_internals.h +++ b/src/include/storage/buf_internals.h @@ -30,7 +30,7 @@ #include "utils/resowner.h" /* - * Buffer state is a single 32-bit variable where following data is combined. + * Buffer state is a single 64-bit variable where following data is combined. * * State of the buffer itself (in order): * - 18 bits refcount @@ -40,6 +40,9 @@ * Combining these values allows to perform some operations without locking * the buffer header, by modifying them together with a CAS loop. * + * NB: A future commit will use a significant portion of the remaining bits to + * implement buffer locking as part of the state variable. + * * The definition of buffer state components is below. */ #define BUF_REFCOUNT_BITS 18 @@ -52,27 +55,27 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, /* refcount related definitions */ #define BUF_REFCOUNT_ONE 1 #define BUF_REFCOUNT_MASK \ - ((1U << BUF_REFCOUNT_BITS) - 1) + ((UINT64CONST(1) << BUF_REFCOUNT_BITS) - 1) /* usage count related definitions */ #define BUF_USAGECOUNT_SHIFT \ BUF_REFCOUNT_BITS #define BUF_USAGECOUNT_MASK \ - (((1U << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) + (((UINT64CONST(1) << BUF_USAGECOUNT_BITS) - 1) << (BUF_USAGECOUNT_SHIFT)) #define BUF_USAGECOUNT_ONE \ - (1U << BUF_REFCOUNT_BITS) + (UINT64CONST(1) << BUF_REFCOUNT_BITS) /* flags related definitions */ #define BUF_FLAG_SHIFT \ (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS) #define BUF_FLAG_MASK \ - (((1U << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) + (((UINT64CONST(1) << BUF_FLAG_BITS) - 1) << BUF_FLAG_SHIFT) /* Get refcount and usagecount from buffer state */ #define BUF_STATE_GET_REFCOUNT(state) \ - ((state) & BUF_REFCOUNT_MASK) + ((uint32)((state) & BUF_REFCOUNT_MASK)) #define BUF_STATE_GET_USAGECOUNT(state) \ - (((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT) + ((uint32)(((state) & BUF_USAGECOUNT_MASK) >> BUF_USAGECOUNT_SHIFT)) /* * Flags for buffer descriptors @@ -82,7 +85,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BUF_DEFINE_FLAG(flagno) \ - (1U << (BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + (flagno))) + (UINT64CONST(1) << (BUF_FLAG_SHIFT + (flagno))) /* buffer header is locked */ #define BM_LOCKED BUF_DEFINE_FLAG( 0) @@ -115,7 +118,7 @@ StaticAssertDecl(BUF_REFCOUNT_BITS + BUF_USAGECOUNT_BITS + BUF_FLAG_BITS == 32, */ #define BM_MAX_USAGE_COUNT 5 -StaticAssertDecl(BM_MAX_USAGE_COUNT < (1 << BUF_USAGECOUNT_BITS), +StaticAssertDecl(BM_MAX_USAGE_COUNT < (UINT64CONST(1) << BUF_USAGECOUNT_BITS), "BM_MAX_USAGE_COUNT doesn't fit in BUF_USAGECOUNT_BITS bits"); StaticAssertDecl(MAX_BACKENDS_BITS <= BUF_REFCOUNT_BITS, "MAX_BACKENDS_BITS needs to be <= BUF_REFCOUNT_BITS"); @@ -280,8 +283,8 @@ BufMappingPartitionLockByIndex(uint32 index) * We use this same struct for local buffer headers, but the locks are not * used and not all of the flag bits are useful either. To avoid unnecessary * overhead, manipulations of the state field should be done without actual - * atomic operations (i.e. only pg_atomic_read_u32() and - * pg_atomic_unlocked_write_u32()). + * atomic operations (i.e. only pg_atomic_read_u64() and + * pg_atomic_unlocked_write_u64()). * * Be careful to avoid increasing the size of the struct when adding or * reordering members. Keeping it below 64 bytes (the most common CPU @@ -309,7 +312,7 @@ typedef struct BufferDesc * State of the buffer, containing flags, refcount and usagecount. See * BUF_* and BM_* defines at the top of this file. */ - pg_atomic_uint32 state; + pg_atomic_uint64 state; /* * Backend of pin-count waiter. The buffer header spinlock needs to be @@ -415,7 +418,7 @@ BufferDescriptorGetContentLock(const BufferDesc *bdesc) * Functions for acquiring/releasing a shared buffer header's spinlock. Do * not apply these to local buffers! */ -extern uint32 LockBufHdr(BufferDesc *desc); +extern uint64 LockBufHdr(BufferDesc *desc); /* * Unlock the buffer header. @@ -426,9 +429,9 @@ extern uint32 LockBufHdr(BufferDesc *desc); static inline void UnlockBufHdr(BufferDesc *desc) { - Assert(pg_atomic_read_u32(&desc->state) & BM_LOCKED); + Assert(pg_atomic_read_u64(&desc->state) & BM_LOCKED); - pg_atomic_fetch_sub_u32(&desc->state, BM_LOCKED); + pg_atomic_fetch_sub_u64(&desc->state, BM_LOCKED); } /* @@ -439,14 +442,14 @@ UnlockBufHdr(BufferDesc *desc) * Note that this approach would not work for usagecount, since we need to cap * the usagecount at BM_MAX_USAGE_COUNT. */ -static inline uint32 -UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, - uint32 set_bits, uint32 unset_bits, +static inline uint64 +UnlockBufHdrExt(BufferDesc *desc, uint64 old_buf_state, + uint64 set_bits, uint64 unset_bits, int refcount_change) { for (;;) { - uint32 buf_state = old_buf_state; + uint64 buf_state = old_buf_state; Assert(buf_state & BM_LOCKED); @@ -457,7 +460,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, if (refcount_change != 0) buf_state += BUF_REFCOUNT_ONE * refcount_change; - if (pg_atomic_compare_exchange_u32(&desc->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&desc->state, &old_buf_state, buf_state)) { return old_buf_state; @@ -465,7 +468,7 @@ UnlockBufHdrExt(BufferDesc *desc, uint32 old_buf_state, } } -extern uint32 WaitBufHdrUnlocked(BufferDesc *buf); +extern uint64 WaitBufHdrUnlocked(BufferDesc *buf); /* in bufmgr.c */ @@ -525,14 +528,14 @@ extern void TrackNewBufferPin(Buffer buf); /* solely to make it easier to write tests */ extern bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait); -extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +extern void TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio); /* freelist.c */ extern IOContext IOContextForStrategy(BufferAccessStrategy strategy); extern BufferDesc *StrategyGetBuffer(BufferAccessStrategy strategy, - uint32 *buf_state, bool *from_ring); + uint64 *buf_state, bool *from_ring); extern bool StrategyRejectBuffer(BufferAccessStrategy strategy, BufferDesc *buf, bool from_ring); @@ -568,7 +571,7 @@ extern BlockNumber ExtendBufferedRelLocal(BufferManagerRelation bmr, uint32 *extended_by); extern void MarkLocalBufferDirty(Buffer buffer); extern void TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, - uint32 set_flag_bits, bool release_aio); + uint64 set_flag_bits, bool release_aio); extern bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait); extern void FlushLocalBuffer(BufferDesc *bufHdr, SMgrRelation reln); extern void InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced); diff --git a/src/include/storage/procnumber.h b/src/include/storage/procnumber.h index 30c360ad350..bd9cb3891cc 100644 --- a/src/include/storage/procnumber.h +++ b/src/include/storage/procnumber.h @@ -27,13 +27,13 @@ typedef int ProcNumber; /* * Note: MAX_BACKENDS_BITS is 18 as that is the space available for buffer - * refcounts in buf_internals.h. This limitation could be lifted by using a - * 64bit state; but it's unlikely to be worthwhile as 2^18-1 backends exceed - * currently realistic configurations. Even if that limitation were removed, - * we still could not a) exceed 2^23-1 because inval.c stores the ProcNumber - * as a 3-byte signed integer, b) INT_MAX/4 because some places compute - * 4*MaxBackends without any overflow check. We check that the configured - * number of backends does not exceed MAX_BACKENDS in InitializeMaxBackends(). + * refcounts in buf_internals.h. This limitation could be lifted, but it's + * unlikely to be worthwhile as 2^18-1 backends exceed currently realistic + * configurations. Even if that limitation were removed, we still could not a) + * exceed 2^23-1 because inval.c stores the ProcNumber as a 3-byte signed + * integer, b) INT_MAX/4 because some places compute 4*MaxBackends without any + * overflow check. We check that the configured number of backends does not + * exceed MAX_BACKENDS in InitializeMaxBackends(). */ #define MAX_BACKENDS_BITS 18 #define MAX_BACKENDS ((1U << MAX_BACKENDS_BITS)-1) diff --git a/src/backend/storage/buffer/buf_init.c b/src/backend/storage/buffer/buf_init.c index 9a312bcc7b3..7d894522526 100644 --- a/src/backend/storage/buffer/buf_init.c +++ b/src/backend/storage/buffer/buf_init.c @@ -121,7 +121,7 @@ BufferManagerShmemInit(void) ClearBufferTag(&buf->tag); - pg_atomic_init_u32(&buf->state, 0); + pg_atomic_init_u64(&buf->state, 0); buf->wait_backend_pgprocno = INVALID_PROC_NUMBER; buf->buf_id = i; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index a036c2aa275..b0de8e45d4d 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -780,7 +780,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN { BufferDesc *bufHdr; BufferTag tag; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(recent_buffer)); @@ -793,7 +793,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN int b = -recent_buffer - 1; bufHdr = GetLocalBufferDescriptor(b); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* Is it still valid and holding the right tag? */ if ((buf_state & BM_VALID) && BufferTagsEqual(&tag, &bufHdr->tag)) @@ -1386,8 +1386,8 @@ StartReadBuffersImpl(ReadBuffersOperation *operation, bufHdr = GetLocalBufferDescriptor(-buffers[i] - 1); else bufHdr = GetBufferDescriptor(buffers[i] - 1); - Assert(pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID); - found = pg_atomic_read_u32(&bufHdr->state) & BM_VALID; + Assert(pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID); + found = pg_atomic_read_u64(&bufHdr->state) & BM_VALID; } else { @@ -1613,10 +1613,10 @@ CheckReadBuffersOperation(ReadBuffersOperation *operation, bool is_complete) GetBufferDescriptor(buffer - 1); Assert(BufferGetBlockNumber(buffer) == operation->blocknum + i); - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_TAG_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_TAG_VALID); if (i < operation->nblocks_done) - Assert(pg_atomic_read_u32(&buf_hdr->state) & BM_VALID); + Assert(pg_atomic_read_u64(&buf_hdr->state) & BM_VALID); } #endif } @@ -2083,8 +2083,8 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, int existing_buf_id; Buffer victim_buffer; BufferDesc *victim_buf_hdr; - uint32 victim_buf_state; - uint32 set_bits = 0; + uint64 victim_buf_state; + uint64 set_bits = 0; /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlarge(CurrentResourceOwner); @@ -2251,7 +2251,7 @@ InvalidateBuffer(BufferDesc *buf) uint32 oldHash; /* hash value for oldTag */ LWLock *oldPartitionLock; /* buffer partition lock for it */ uint32 oldFlags; - uint32 buf_state; + uint64 buf_state; /* Save the original buffer tag before dropping the spinlock */ oldTag = buf->tag; @@ -2342,7 +2342,7 @@ retry: static bool InvalidateVictimBuffer(BufferDesc *buf_hdr) { - uint32 buf_state; + uint64 buf_state; uint32 hash; LWLock *partition_lock; BufferTag tag; @@ -2402,10 +2402,10 @@ InvalidateVictimBuffer(BufferDesc *buf_hdr) LWLockRelease(partition_lock); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(!(buf_state & (BM_DIRTY | BM_VALID | BM_TAG_VALID))); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); - Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u32(&buf_hdr->state)) > 0); + Assert(BUF_STATE_GET_REFCOUNT(pg_atomic_read_u64(&buf_hdr->state)) > 0); return true; } @@ -2415,7 +2415,7 @@ GetVictimBuffer(BufferAccessStrategy strategy, IOContext io_context) { BufferDesc *buf_hdr; Buffer buf; - uint32 buf_state; + uint64 buf_state; bool from_ring; /* @@ -2548,7 +2548,7 @@ again: /* a final set of sanity checks */ #ifdef USE_ASSERT_CHECKING - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) == 1); Assert(!(buf_state & (BM_TAG_VALID | BM_VALID | BM_DIRTY))); @@ -2839,13 +2839,13 @@ ExtendBufferedRelShared(BufferManagerRelation bmr, */ do { - pg_atomic_fetch_and_u32(&existing_hdr->state, ~BM_VALID); + pg_atomic_fetch_and_u64(&existing_hdr->state, ~BM_VALID); } while (!StartBufferIO(existing_hdr, true, false)); } else { - uint32 buf_state; - uint32 set_bits = 0; + uint64 buf_state; + uint64 set_bits = 0; buf_state = LockBufHdr(victim_buf_hdr); @@ -3021,7 +3021,7 @@ BufferIsDirty(Buffer buffer) Assert(BufferIsLockedByMeInMode(buffer, BUFFER_LOCK_EXCLUSIVE)); } - return pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY; + return pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY; } /* @@ -3037,8 +3037,8 @@ void MarkBufferDirty(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; if (!BufferIsValid(buffer)) elog(ERROR, "bad buffer ID: %d", buffer); @@ -3058,7 +3058,7 @@ MarkBufferDirty(Buffer buffer) * NB: We have to wait for the buffer header spinlock to be not held, as * TerminateBufferIO() relies on the spinlock. */ - old_buf_state = pg_atomic_read_u32(&bufHdr->state); + old_buf_state = pg_atomic_read_u64(&bufHdr->state); for (;;) { if (old_buf_state & BM_LOCKED) @@ -3069,7 +3069,7 @@ MarkBufferDirty(Buffer buffer) Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state |= BM_DIRTY | BM_JUST_DIRTIED; - if (pg_atomic_compare_exchange_u32(&bufHdr->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&bufHdr->state, &old_buf_state, buf_state)) break; } @@ -3173,10 +3173,10 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, if (ref == NULL) { - uint32 buf_state; - uint32 old_buf_state; + uint64 buf_state; + uint64 old_buf_state; - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { if (unlikely(skip_if_not_valid && !(old_buf_state & BM_VALID))) @@ -3210,7 +3210,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, buf_state += BUF_USAGECOUNT_ONE; } - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, buf_state)) { result = (buf_state & BM_VALID) != 0; @@ -3237,7 +3237,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, * that the buffer page is legitimately non-accessible here. We * cannot meddle with that. */ - result = (pg_atomic_read_u32(&buf->state) & BM_VALID) != 0; + result = (pg_atomic_read_u64(&buf->state) & BM_VALID) != 0; Assert(ref->data.refcount > 0); ref->data.refcount++; @@ -3272,7 +3272,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy, static void PinBuffer_Locked(BufferDesc *buf) { - uint32 old_buf_state; + uint64 old_buf_state; /* * As explained, We don't expect any preexisting pins. That allows us to @@ -3284,7 +3284,7 @@ PinBuffer_Locked(BufferDesc *buf) * Since we hold the buffer spinlock, we can update the buffer state and * release the lock in one operation. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); UnlockBufHdrExt(buf, old_buf_state, 0, 0, 1); @@ -3314,7 +3314,7 @@ WakePinCountWaiter(BufferDesc *buf) * BM_PIN_COUNT_WAITER if it stops waiting for a reason other than this * backend waking it up. */ - uint32 buf_state = LockBufHdr(buf); + uint64 buf_state = LockBufHdr(buf); if ((buf_state & BM_PIN_COUNT_WAITER) && BUF_STATE_GET_REFCOUNT(buf_state) == 1) @@ -3361,7 +3361,7 @@ UnpinBufferNoOwner(BufferDesc *buf) ref->data.refcount--; if (ref->data.refcount == 0) { - uint32 old_buf_state; + uint64 old_buf_state; /* * Mark buffer non-accessible to Valgrind. @@ -3379,7 +3379,7 @@ UnpinBufferNoOwner(BufferDesc *buf) Assert(!LWLockHeldByMe(BufferDescriptorGetContentLock(buf))); /* decrement the shared reference count */ - old_buf_state = pg_atomic_fetch_sub_u32(&buf->state, BUF_REFCOUNT_ONE); + old_buf_state = pg_atomic_fetch_sub_u64(&buf->state, BUF_REFCOUNT_ONE); /* Support LockBufferForCleanup() */ if (old_buf_state & BM_PIN_COUNT_WAITER) @@ -3436,7 +3436,7 @@ TrackNewBufferPin(Buffer buf) static void BufferSync(int flags) { - uint32 buf_state; + uint64 buf_state; int buf_id; int num_to_scan; int num_spaces; @@ -3446,7 +3446,7 @@ BufferSync(int flags) Oid last_tsid; binaryheap *ts_heap; int i; - uint32 mask = BM_DIRTY; + uint64 mask = BM_DIRTY; WritebackContext wb_context; /* @@ -3478,7 +3478,7 @@ BufferSync(int flags) for (buf_id = 0; buf_id < NBuffers; buf_id++) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); - uint32 set_bits = 0; + uint64 set_bits = 0; /* * Header spinlock is enough to examine BM_DIRTY, see comment in @@ -3645,7 +3645,7 @@ BufferSync(int flags) * write the buffer though we didn't need to. It doesn't seem worth * guarding against this, though. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_CHECKPOINT_NEEDED) + if (pg_atomic_read_u64(&bufHdr->state) & BM_CHECKPOINT_NEEDED) { if (SyncOneBuffer(buf_id, false, &wb_context) & BUF_WRITTEN) { @@ -4015,7 +4015,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) { BufferDesc *bufHdr = GetBufferDescriptor(buf_id); int result = 0; - uint32 buf_state; + uint64 buf_state; BufferTag tag; /* Make sure we can handle the pin */ @@ -4264,7 +4264,7 @@ DebugPrintBufferRefcount(Buffer buffer) int32 loccount; char *result; ProcNumber backend; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); if (BufferIsLocal(buffer)) @@ -4281,9 +4281,9 @@ DebugPrintBufferRefcount(Buffer buffer) } /* theoretically we should lock the bufHdr here */ - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); - result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%x, refcount=%u %d)", + result = psprintf("[%03d] (rel=%s, blockNum=%u, flags=0x%" PRIx64 ", refcount=%u %d)", buffer, relpathbackend(BufTagGetRelFileLocator(&buf->tag), backend, BufTagGetForkNum(&buf->tag)).str, @@ -4383,7 +4383,7 @@ FlushBuffer(BufferDesc *buf, SMgrRelation reln, IOObject io_object, instr_time io_start; Block bufBlock; char *bufToWrite; - uint32 buf_state; + uint64 buf_state; /* * Try to start an I/O operation. If StartBufferIO returns false, then @@ -4581,7 +4581,7 @@ BufferIsPermanent(Buffer buffer) * not random garbage. */ bufHdr = GetBufferDescriptor(buffer - 1); - return (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT) != 0; + return (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT) != 0; } /* @@ -5044,11 +5044,11 @@ FlushRelationBuffers(Relation rel) { for (i = 0; i < NLocBuffer; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetLocalBufferDescriptor(i); if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) && - ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & + ((buf_state = pg_atomic_read_u64(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { ErrorContextCallback errcallback; @@ -5084,7 +5084,7 @@ FlushRelationBuffers(Relation rel) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5156,7 +5156,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) { SMgrSortArray *srelent = NULL; BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; /* * As in DropRelationBuffers, an unlocked precheck should be safe and @@ -5405,7 +5405,7 @@ FlushDatabaseBuffers(Oid dbid) for (i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; bufHdr = GetBufferDescriptor(i); @@ -5553,13 +5553,13 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * is only intended to be used in cases where failing to write out the * data would be harmless anyway, it doesn't really matter. */ - if ((pg_atomic_read_u32(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != + if ((pg_atomic_read_u64(&bufHdr->state) & (BM_DIRTY | BM_JUST_DIRTIED)) != (BM_DIRTY | BM_JUST_DIRTIED)) { XLogRecPtr lsn = InvalidXLogRecPtr; bool dirtied = false; bool delayChkptFlags = false; - uint32 buf_state; + uint64 buf_state; /* * If we need to protect hint bit updates from torn writes, WAL-log a @@ -5571,7 +5571,7 @@ MarkBufferDirtyHint(Buffer buffer, bool buffer_std) * when we call XLogInsert() since the value changes dynamically. */ if (XLogHintBitIsNeeded() && - (pg_atomic_read_u32(&bufHdr->state) & BM_PERMANENT)) + (pg_atomic_read_u64(&bufHdr->state) & BM_PERMANENT)) { /* * If we must not write WAL, due to a relfilelocator-specific @@ -5671,8 +5671,8 @@ UnlockBuffers(void) if (buf) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; buf_state = LockBufHdr(buf); @@ -5803,8 +5803,8 @@ LockBufferForCleanup(Buffer buffer) for (;;) { - uint32 buf_state; - uint32 unset_bits = 0; + uint64 buf_state; + uint64 unset_bits = 0; /* Try to acquire lock */ LockBuffer(buffer, BUFFER_LOCK_EXCLUSIVE); @@ -5952,7 +5952,7 @@ bool ConditionalLockBufferForCleanup(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state, + uint64 buf_state, refcount; Assert(BufferIsValid(buffer)); @@ -6010,7 +6010,7 @@ bool IsBufferCleanupOK(Buffer buffer) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsValid(buffer)); @@ -6066,7 +6066,7 @@ WaitIO(BufferDesc *buf) ConditionVariablePrepareToSleep(cv); for (;;) { - uint32 buf_state; + uint64 buf_state; PgAioWaitRef iow; /* @@ -6140,7 +6140,7 @@ WaitIO(BufferDesc *buf) bool StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; ResourceOwnerEnlarge(CurrentResourceOwner); @@ -6196,11 +6196,11 @@ StartBufferIO(BufferDesc *buf, bool forInput, bool nowait) * is being released) */ void -TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint32 set_flag_bits, +TerminateBufferIO(BufferDesc *buf, bool clear_dirty, uint64 set_flag_bits, bool forget_owner, bool release_aio) { - uint32 buf_state; - uint32 unset_flag_bits = 0; + uint64 buf_state; + uint64 unset_flag_bits = 0; int refcount_change = 0; buf_state = LockBufHdr(buf); @@ -6261,7 +6261,7 @@ static void AbortBufferIO(Buffer buffer) { BufferDesc *buf_hdr = GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; buf_state = LockBufHdr(buf_hdr); Assert(buf_state & (BM_IO_IN_PROGRESS | BM_TAG_VALID)); @@ -6355,10 +6355,10 @@ rlocator_comparator(const void *p1, const void *p2) /* * Lock buffer header - set BM_LOCKED in buffer state. */ -uint32 +uint64 LockBufHdr(BufferDesc *desc) { - uint32 old_buf_state; + uint64 old_buf_state; Assert(!BufferIsLocal(BufferDescriptorGetBuffer(desc))); @@ -6369,7 +6369,7 @@ LockBufHdr(BufferDesc *desc) * the spin-delay infrastructure. The work necessary for that shows up * in profiles and is rarely necessary. */ - old_buf_state = pg_atomic_fetch_or_u32(&desc->state, BM_LOCKED); + old_buf_state = pg_atomic_fetch_or_u64(&desc->state, BM_LOCKED); if (likely(!(old_buf_state & BM_LOCKED))) break; /* got lock */ @@ -6382,7 +6382,7 @@ LockBufHdr(BufferDesc *desc) while (old_buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - old_buf_state = pg_atomic_read_u32(&desc->state); + old_buf_state = pg_atomic_read_u64(&desc->state); } finish_spin_delay(&delayStatus); } @@ -6403,20 +6403,20 @@ LockBufHdr(BufferDesc *desc) * Obviously the buffer could be locked by the time the value is returned, so * this is primarily useful in CAS style loops. */ -pg_noinline uint32 +pg_noinline uint64 WaitBufHdrUnlocked(BufferDesc *buf) { SpinDelayStatus delayStatus; - uint32 buf_state; + uint64 buf_state; init_local_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); while (buf_state & BM_LOCKED) { perform_spin_delay(&delayStatus); - buf_state = pg_atomic_read_u32(&buf->state); + buf_state = pg_atomic_read_u64(&buf->state); } finish_spin_delay(&delayStatus); @@ -6704,12 +6704,12 @@ ResOwnerPrintBufferPin(Datum res) static bool EvictUnpinnedBufferInternal(BufferDesc *desc, bool *buffer_flushed) { - uint32 buf_state; + uint64 buf_state; bool result; *buffer_flushed = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -6803,12 +6803,12 @@ EvictAllUnpinnedBuffers(int32 *buffers_evicted, int32 *buffers_flushed, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_flushed; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -6855,7 +6855,7 @@ EvictRelUnpinnedBuffers(Relation rel, int32 *buffers_evicted, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_flushed; CHECK_FOR_INTERRUPTS(); @@ -6897,12 +6897,12 @@ static bool MarkDirtyUnpinnedBufferInternal(Buffer buf, BufferDesc *desc, bool *buffer_already_dirty) { - uint32 buf_state; + uint64 buf_state; bool result = false; *buffer_already_dirty = false; - buf_state = pg_atomic_read_u32(&(desc->state)); + buf_state = pg_atomic_read_u64(&(desc->state)); Assert(buf_state & BM_LOCKED); if ((buf_state & BM_VALID) == 0) @@ -7000,7 +7000,7 @@ MarkDirtyRelUnpinnedBuffers(Relation rel, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state = pg_atomic_read_u32(&(desc->state)); + uint64 buf_state = pg_atomic_read_u64(&(desc->state)); bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); @@ -7054,12 +7054,12 @@ MarkDirtyAllUnpinnedBuffers(int32 *buffers_dirtied, for (int buf = 1; buf <= NBuffers; buf++) { BufferDesc *desc = GetBufferDescriptor(buf - 1); - uint32 buf_state; + uint64 buf_state; bool buffer_already_dirty; CHECK_FOR_INTERRUPTS(); - buf_state = pg_atomic_read_u32(&desc->state); + buf_state = pg_atomic_read_u64(&desc->state); if (!(buf_state & BM_VALID)) continue; @@ -7110,7 +7110,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) BufferDesc *buf_hdr = is_temp ? GetLocalBufferDescriptor(-buffer - 1) : GetBufferDescriptor(buffer - 1); - uint32 buf_state; + uint64 buf_state; /* * Check that all the buffers are actually ones that could conceivably @@ -7128,7 +7128,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) } if (is_temp) - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); else buf_state = LockBufHdr(buf_hdr); @@ -7166,7 +7166,7 @@ buffer_stage_common(PgAioHandle *ioh, bool is_write, bool is_temp) if (is_temp) { buf_state += BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else UnlockBufHdrExt(buf_hdr, buf_state, 0, 0, 1); @@ -7352,13 +7352,13 @@ buffer_readv_complete_one(PgAioTargetData *td, uint8 buf_off, Buffer buffer, : GetBufferDescriptor(buffer - 1); BufferTag tag = buf_hdr->tag; char *bufdata = BufferGetBlock(buffer); - uint32 set_flag_bits; + uint64 set_flag_bits; int piv_flags; /* check that the buffer is in the expected state for a read */ #ifdef USE_ASSERT_CHECKING { - uint32 buf_state = pg_atomic_read_u32(&buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_VALID)); diff --git a/src/backend/storage/buffer/freelist.c b/src/backend/storage/buffer/freelist.c index 9a93fb335fc..b7687836188 100644 --- a/src/backend/storage/buffer/freelist.c +++ b/src/backend/storage/buffer/freelist.c @@ -86,7 +86,7 @@ typedef struct BufferAccessStrategyData /* Prototypes for internal functions */ static BufferDesc *GetBufferFromRing(BufferAccessStrategy strategy, - uint32 *buf_state); + uint64 *buf_state); static void AddBufferToRing(BufferAccessStrategy strategy, BufferDesc *buf); @@ -171,7 +171,7 @@ ClockSweepTick(void) * before returning. */ BufferDesc * -StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_ring) +StrategyGetBuffer(BufferAccessStrategy strategy, uint64 *buf_state, bool *from_ring) { BufferDesc *buf; int bgwprocno; @@ -230,8 +230,8 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r trycounter = NBuffers; for (;;) { - uint32 old_buf_state; - uint32 local_buf_state; + uint64 old_buf_state; + uint64 local_buf_state; buf = GetBufferDescriptor(ClockSweepTick()); @@ -239,7 +239,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r * Check whether the buffer can be used and pin it if so. Do this * using a CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -277,7 +277,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r { local_buf_state -= BUF_USAGECOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { trycounter = NBuffers; @@ -289,7 +289,7 @@ StrategyGetBuffer(BufferAccessStrategy strategy, uint32 *buf_state, bool *from_r /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { /* Found a usable buffer */ @@ -655,12 +655,12 @@ FreeAccessStrategy(BufferAccessStrategy strategy) * returning. */ static BufferDesc * -GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) +GetBufferFromRing(BufferAccessStrategy strategy, uint64 *buf_state) { BufferDesc *buf; Buffer bufnum; - uint32 old_buf_state; - uint32 local_buf_state; /* to avoid repeated (de-)referencing */ + uint64 old_buf_state; + uint64 local_buf_state; /* to avoid repeated (de-)referencing */ /* Advance to next ring slot */ @@ -682,7 +682,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) * Check whether the buffer can be used and pin it if so. Do this using a * CAS loop, to avoid having to lock the buffer header. */ - old_buf_state = pg_atomic_read_u32(&buf->state); + old_buf_state = pg_atomic_read_u64(&buf->state); for (;;) { local_buf_state = old_buf_state; @@ -710,7 +710,7 @@ GetBufferFromRing(BufferAccessStrategy strategy, uint32 *buf_state) /* pin the buffer if the CAS succeeds */ local_buf_state += BUF_REFCOUNT_ONE; - if (pg_atomic_compare_exchange_u32(&buf->state, &old_buf_state, + if (pg_atomic_compare_exchange_u64(&buf->state, &old_buf_state, local_buf_state)) { *buf_state = local_buf_state; diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index f6e2b1aa288..04a540379a2 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -148,7 +148,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, } else { - uint32 buf_state; + uint64 buf_state; victim_buffer = GetLocalVictimBuffer(); bufid = -victim_buffer - 1; @@ -165,10 +165,10 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, */ bufHdr->tag = newTag; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); *foundPtr = false; } @@ -245,12 +245,12 @@ GetLocalVictimBuffer(void) if (LocalRefCount[victim_bufid] == 0) { - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); if (BUF_STATE_GET_USAGECOUNT(buf_state) > 0) { buf_state -= BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); trycounter = NLocBuffer; } else if (BUF_STATE_GET_REFCOUNT(buf_state) > 0) @@ -286,13 +286,13 @@ GetLocalVictimBuffer(void) * this buffer is not referenced but it might still be dirty. if that's * the case, write it out before reusing it! */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&bufHdr->state) & BM_DIRTY) FlushLocalBuffer(bufHdr, NULL); /* * Remove the victim buffer from the hashtable and mark as invalid. */ - if (pg_atomic_read_u32(&bufHdr->state) & BM_TAG_VALID) + if (pg_atomic_read_u64(&bufHdr->state) & BM_TAG_VALID) { InvalidateLocalBuffer(bufHdr, false); @@ -417,7 +417,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, if (found) { BufferDesc *existing_hdr; - uint32 buf_state; + uint64 buf_state; UnpinLocalBuffer(BufferDescriptorGetBuffer(victim_buf_hdr)); @@ -428,18 +428,18 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, /* * Clear the BM_VALID bit, do StartLocalBufferIO() and proceed. */ - buf_state = pg_atomic_read_u32(&existing_hdr->state); + buf_state = pg_atomic_read_u64(&existing_hdr->state); Assert(buf_state & BM_TAG_VALID); Assert(!(buf_state & BM_DIRTY)); buf_state &= ~BM_VALID; - pg_atomic_unlocked_write_u32(&existing_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&existing_hdr->state, buf_state); /* no need to loop for local buffers */ StartLocalBufferIO(existing_hdr, true, false); } else { - uint32 buf_state = pg_atomic_read_u32(&victim_buf_hdr->state); + uint64 buf_state = pg_atomic_read_u64(&victim_buf_hdr->state); Assert(!(buf_state & (BM_VALID | BM_TAG_VALID | BM_DIRTY | BM_JUST_DIRTIED))); @@ -447,7 +447,7 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, buf_state |= BM_TAG_VALID | BUF_USAGECOUNT_ONE; - pg_atomic_unlocked_write_u32(&victim_buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&victim_buf_hdr->state, buf_state); hresult->id = victim_buf_id; @@ -467,13 +467,13 @@ ExtendBufferedRelLocal(BufferManagerRelation bmr, { Buffer buf = buffers[i]; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); buf_state |= BM_VALID; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } *extended_by = extend_by; @@ -492,7 +492,7 @@ MarkLocalBufferDirty(Buffer buffer) { int bufid; BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; Assert(BufferIsLocal(buffer)); @@ -506,14 +506,14 @@ MarkLocalBufferDirty(Buffer buffer) bufHdr = GetLocalBufferDescriptor(bufid); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_DIRTY)) pgBufferUsage.local_blks_dirtied++; buf_state |= BM_DIRTY; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -522,7 +522,7 @@ MarkLocalBufferDirty(Buffer buffer) bool StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) { - uint32 buf_state; + uint64 buf_state; /* * With AIO the buffer could have IO in progress, e.g. when there are two @@ -542,7 +542,7 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) /* Once we get here, there is definitely no I/O active on this buffer */ /* Check if someone else already did the I/O */ - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (forInput ? (buf_state & BM_VALID) : !(buf_state & BM_DIRTY)) { return false; @@ -559,11 +559,11 @@ StartLocalBufferIO(BufferDesc *bufHdr, bool forInput, bool nowait) * Like TerminateBufferIO, but for local buffers */ void -TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bits, +TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint64 set_flag_bits, bool release_aio) { /* Only need to adjust flags */ - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); /* BM_IO_IN_PROGRESS isn't currently used for local buffers */ @@ -582,7 +582,7 @@ TerminateLocalBufferIO(BufferDesc *bufHdr, bool clear_dirty, uint32 set_flag_bit } buf_state |= set_flag_bits; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); /* local buffers don't track IO using resowners */ @@ -606,7 +606,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) { Buffer buffer = BufferDescriptorGetBuffer(bufHdr); int bufid = -buffer - 1; - uint32 buf_state; + uint64 buf_state; LocalBufferLookupEnt *hresult; /* @@ -622,7 +622,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) Assert(!pgaio_wref_valid(&bufHdr->io_wref)); } - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); /* * We need to test not just LocalRefCount[bufid] but also the BufferDesc @@ -647,7 +647,7 @@ InvalidateLocalBuffer(BufferDesc *bufHdr, bool check_unreferenced) ClearBufferTag(&bufHdr->tag); buf_state &= ~BUF_FLAG_MASK; buf_state &= ~BUF_USAGECOUNT_MASK; - pg_atomic_unlocked_write_u32(&bufHdr->state, buf_state); + pg_atomic_unlocked_write_u64(&bufHdr->state, buf_state); } /* @@ -671,9 +671,9 @@ DropRelationLocalBuffers(RelFileLocator rlocator, ForkNumber *forkNum, for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (!(buf_state & BM_TAG_VALID) || !BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -706,9 +706,9 @@ DropRelationAllLocalBuffers(RelFileLocator rlocator) for (i = 0; i < NLocBuffer; i++) { BufferDesc *bufHdr = GetLocalBufferDescriptor(i); - uint32 buf_state; + uint64 buf_state; - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if ((buf_state & BM_TAG_VALID) && BufTagMatchesRelFileLocator(&bufHdr->tag, &rlocator)) @@ -804,11 +804,11 @@ InitLocalBuffers(void) bool PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { - uint32 buf_state; + uint64 buf_state; Buffer buffer = BufferDescriptorGetBuffer(buf_hdr); int bufid = -buffer - 1; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); if (LocalRefCount[bufid] == 0) { @@ -819,7 +819,7 @@ PinLocalBuffer(BufferDesc *buf_hdr, bool adjust_usagecount) { buf_state += BUF_USAGECOUNT_ONE; } - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* * See comment in PinBuffer(). @@ -856,14 +856,14 @@ UnpinLocalBufferNoOwner(Buffer buffer) if (--LocalRefCount[buffid] == 0) { BufferDesc *buf_hdr = GetLocalBufferDescriptor(buffid); - uint32 buf_state; + uint64 buf_state; NLocalPinnedBuffers--; - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); Assert(BUF_STATE_GET_REFCOUNT(buf_state) > 0); buf_state -= BUF_REFCOUNT_ONE; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); /* see comment in UnpinBufferNoOwner */ VALGRIND_MAKE_MEM_NOACCESS(LocalBufHdrGetBlock(buf_hdr), BLCKSZ); diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index b682dca658b..dcba3fb5473 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -199,7 +199,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) for (i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -615,7 +615,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr; - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); @@ -626,7 +626,7 @@ pg_buffercache_summary(PG_FUNCTION_ARGS) * noticeably increase the cost of the function. */ bufHdr = GetBufferDescriptor(i); - buf_state = pg_atomic_read_u32(&bufHdr->state); + buf_state = pg_atomic_read_u64(&bufHdr->state); if (buf_state & BM_VALID) { @@ -676,7 +676,7 @@ pg_buffercache_usage_counts(PG_FUNCTION_ARGS) for (int i = 0; i < NBuffers; i++) { BufferDesc *bufHdr = GetBufferDescriptor(i); - uint32 buf_state = pg_atomic_read_u32(&bufHdr->state); + uint64 buf_state = pg_atomic_read_u64(&bufHdr->state); int usage_count; CHECK_FOR_INTERRUPTS(); diff --git a/contrib/pg_prewarm/autoprewarm.c b/contrib/pg_prewarm/autoprewarm.c index 3ca7d2ed772..89e187425cc 100644 --- a/contrib/pg_prewarm/autoprewarm.c +++ b/contrib/pg_prewarm/autoprewarm.c @@ -703,7 +703,7 @@ apw_dump_now(bool is_bgworker, bool dump_unlogged) for (num_blocks = 0, i = 0; i < NBuffers; i++) { - uint32 buf_state; + uint64 buf_state; CHECK_FOR_INTERRUPTS(); diff --git a/src/test/modules/test_aio/test_aio.c b/src/test/modules/test_aio/test_aio.c index e046b08f3d5..b1aa8af9ec0 100644 --- a/src/test/modules/test_aio/test_aio.c +++ b/src/test/modules/test_aio/test_aio.c @@ -308,9 +308,9 @@ create_toy_buffer(Relation rel, BlockNumber blkno) { Buffer buf; BufferDesc *buf_hdr; - uint32 buf_state; + uint64 buf_state; bool was_pinned = false; - uint32 unset_bits = 0; + uint64 unset_bits = 0; /* place buffer in shared buffers without erroring out */ buf = ReadBufferExtended(rel, MAIN_FORKNUM, blkno, RBM_ZERO_AND_LOCK, NULL); @@ -319,7 +319,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_hdr = GetLocalBufferDescriptor(-buf - 1); - buf_state = pg_atomic_read_u32(&buf_hdr->state); + buf_state = pg_atomic_read_u64(&buf_hdr->state); } else { @@ -340,7 +340,7 @@ create_toy_buffer(Relation rel, BlockNumber blkno) if (RelationUsesLocalBuffers(rel)) { buf_state &= ~unset_bits; - pg_atomic_unlocked_write_u32(&buf_hdr->state, buf_state); + pg_atomic_unlocked_write_u64(&buf_hdr->state, buf_state); } else { @@ -489,7 +489,7 @@ invalidate_rel_block(PG_FUNCTION_ARGS) LockBuffer(buf, BUFFER_LOCK_EXCLUSIVE); - if (pg_atomic_read_u32(&buf_hdr->state) & BM_DIRTY) + if (pg_atomic_read_u64(&buf_hdr->state) & BM_DIRTY) { if (BufferIsLocal(buf)) FlushLocalBuffer(buf_hdr, NULL); @@ -572,7 +572,7 @@ buffer_call_terminate_io(PG_FUNCTION_ARGS) bool io_error = PG_GETARG_BOOL(3); bool release_aio = PG_GETARG_BOOL(4); bool clear_dirty = false; - uint32 set_flag_bits = 0; + uint64 set_flag_bits = 0; if (io_error) set_flag_bits |= BM_IO_ERROR; -- 2.48.1.76.g4e746b1a31.dirty --c6wqr3sg4llxfu2u Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v11-0002-bufmgr-Implement-buffer-content-locks-independen.patch" ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-01-14 01:10 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-11-04 13:52 [PATCH v10 2/2] handle relation statistics correctly during rewrites Bertrand Drouvot <[email protected]> 2025-11-06 00:22 [PATCH v6 04/14] bufmgr: Change BufferDesc.state to be a 64bit atomic Andres Freund <[email protected]> 2025-12-02 23:46 [PATCH v7 11/15] bufmgr: Change BufferDesc.state to be a 64bit atomic Andres Freund <[email protected]> 2025-12-18 23:28 [PATCH v8 06/10] bufmgr: Change BufferDesc.state to be a 64bit atomic Andres Freund <[email protected]> 2026-01-07 22:26 [PATCH v9 06/10] bufmgr: Change BufferDesc.state to be a 64bit atomic Andres Freund <[email protected]> 2026-01-07 22:26 [PATCH v10 3/8] bufmgr: Change BufferDesc.state to be a 64-bit atomic Andres Freund <[email protected]> 2026-01-14 01:10 [PATCH v11 1/7] bufmgr: Change BufferDesc.state to be a 64-bit atomic Andres Freund <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox