public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 4/8] Add infrastructure to WAL-logging skip feature 3+ messages / 3 participants [nested] [flat]
* [PATCH 4/8] Add infrastructure to WAL-logging skip feature @ 2019-03-26 06:34 Kyotaro Horiguchi <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Kyotaro Horiguchi @ 2019-03-26 06:34 UTC (permalink / raw) We used to optimize WAL-logging for truncation of in-transaction crated tables in minimal mode by just signaling by HEAP_INSERT_SKIP_WAL option on heap operations. This mechanism can emit WAL records that results in corrupt state for certain series of in-transaction operations. This patch provides infrastructure to track pending at-commit fsyncs for a relation and in-transaction truncations. heap_register_sync() should be used to start tracking before batch operations like COPY and CLUSTER, and use BufferNeedsWAL() instead of RelationNeedsWAL() at the places related to WAL-logging about heap-modifying operations. --- src/backend/access/heap/heapam.c | 31 +++ src/backend/access/transam/xact.c | 11 + src/backend/catalog/storage.c | 418 ++++++++++++++++++++++++++++++++++-- src/backend/commands/tablecmds.c | 4 +- src/backend/storage/buffer/bufmgr.c | 39 +++- src/backend/utils/cache/relcache.c | 3 + src/include/access/heapam.h | 1 + src/include/catalog/storage.h | 8 + src/include/storage/bufmgr.h | 2 + src/include/utils/rel.h | 8 + 10 files changed, 493 insertions(+), 32 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index c6e71dba6b..5a8627507f 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -51,6 +51,7 @@ #include "access/xloginsert.h" #include "access/xlogutils.h" #include "catalog/catalog.h" +#include "catalog/storage.h" #include "miscadmin.h" #include "pgstat.h" #include "port/atomics.h" @@ -8800,3 +8801,33 @@ heap_mask(char *pagedata, BlockNumber blkno) } } } + +/* + * heap_register_sync - register a heap to be synced to disk at commit + * + * This can be used to skip WAL-logging changes on a relation file that has + * been created in the same transaction. This makes note of the current size of + * the relation, and ensures that when the relation is extended, any changes + * to the new blocks in the heap, in the same transaction, will not be + * WAL-logged. Instead, the heap contents are flushed to disk at commit, + * like heap_sync() does. + * + * This does the same for the TOAST heap, if any. Indexes are not affected. + */ +void +heap_register_sync(Relation rel) +{ + /* non-WAL-logged tables never need fsync */ + if (!RelationNeedsWAL(rel)) + return; + + RecordWALSkipping(rel); + if (OidIsValid(rel->rd_rel->reltoastrelid)) + { + Relation toastrel; + + toastrel = heap_open(rel->rd_rel->reltoastrelid, AccessShareLock); + RecordWALSkipping(toastrel); + heap_close(toastrel, AccessShareLock); + } +} diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c index c3214d4f4d..ad7cb3bcb9 100644 --- a/src/backend/access/transam/xact.c +++ b/src/backend/access/transam/xact.c @@ -2022,6 +2022,9 @@ CommitTransaction(void) /* close large objects before lower-level cleanup */ AtEOXact_LargeObject(true); + /* Flush updates to relations that we didn't WAL-logged */ + smgrDoPendingSyncs(true); + /* * Mark serializable transaction as complete for predicate locking * purposes. This should be done as late as we can put it and still allow @@ -2254,6 +2257,9 @@ PrepareTransaction(void) /* close large objects before lower-level cleanup */ AtEOXact_LargeObject(true); + /* Flush updates to relations that we didn't WAL-logged */ + smgrDoPendingSyncs(true); + /* * Mark serializable transaction as complete for predicate locking * purposes. This should be done as late as we can put it and still allow @@ -2579,6 +2585,7 @@ AbortTransaction(void) AtAbort_Notify(); AtEOXact_RelationMap(false, is_parallel_worker); AtAbort_Twophase(); + smgrDoPendingSyncs(false); /* abandon pending syncs */ /* * Advertise the fact that we aborted in pg_xact (assuming that we got as @@ -4097,6 +4104,8 @@ ReleaseSavepoint(const char *name) (errcode(ERRCODE_S_E_INVALID_SPECIFICATION), errmsg("savepoint \"%s\" does not exist within current savepoint level", name))); + smgrProcessWALRequirementInval(s->subTransactionId, true); + /* * Mark "commit pending" all subtransactions up to the target * subtransaction. The actual commits will happen when control gets to @@ -4206,6 +4215,8 @@ RollbackToSavepoint(const char *name) (errcode(ERRCODE_S_E_INVALID_SPECIFICATION), errmsg("savepoint \"%s\" does not exist within current savepoint level", name))); + smgrProcessWALRequirementInval(s->subTransactionId, false); + /* * Mark "abort pending" all subtransactions up to the target * subtransaction. The actual aborts will happen when control gets to diff --git a/src/backend/catalog/storage.c b/src/backend/catalog/storage.c index 0302507e6f..be37174ef2 100644 --- a/src/backend/catalog/storage.c +++ b/src/backend/catalog/storage.c @@ -27,7 +27,7 @@ #include "catalog/storage.h" #include "catalog/storage_xlog.h" #include "storage/freespace.h" -#include "storage/smgr.h" +#include "utils/hsearch.h" #include "utils/memutils.h" #include "utils/rel.h" @@ -62,6 +62,58 @@ typedef struct PendingRelDelete static PendingRelDelete *pendingDeletes = NULL; /* head of linked list */ +/* + * We also track relation files (RelFileNode values) that have been created + * in the same transaction, and that have been modified without WAL-logging + * the action (an optimization possible with wal_level=minimal). When we are + * about to skip WAL-logging, a RelWalRequirement entry is created, and + * 'skip_wal_min_blk' is set to the current size of the relation. Any operations + * on blocks < skip_wal_min_blk need to be WAL-logged as usual, but for + * operations on higher blocks, WAL-logging is skipped. + + * + * NB: after WAL-logging has been skipped for a block, we must not WAL-log + * any subsequent actions on the same block either. Replaying the WAL record + * of the subsequent action might fail otherwise, as the "before" state of + * the block might not match, as the earlier actions were not WAL-logged. + * Likewise, after we have WAL-logged an operation for a block, we must + * WAL-log any subsequent operations on the same page as well. Replaying + * a possible full-page-image from the earlier WAL record would otherwise + * revert the page to the old state, even if we sync the relation at end + * of transaction. + * + * If a relation is truncated (without creating a new relfilenode), and we + * emit a WAL record of the truncation, we can't skip WAL-logging for any + * of the truncated blocks anymore, as replaying the truncation record will + * destroy all the data inserted after that. But if we have already decided + * to skip WAL-logging changes to a relation, and the relation is truncated, + * we don't need to WAL-log the truncation either. + * + * This mechanism is currently only used by heaps. Indexes are always + * WAL-logged. Also, this only applies for wal_level=minimal; with higher + * WAL levels we need the WAL for PITR/replication anyway. + */ +typedef struct RelWalRequirement +{ + RelFileNode relnode; /* relation created in same xact */ + bool forks[MAX_FORKNUM + 1]; /* target forknums */ + BlockNumber skip_wal_min_blk; /* WAL-logging skipped for blocks >= + * skip_wal_min_blk */ + BlockNumber wal_log_min_blk; /* The minimum blk number that requires + * WAL-logging even if skipped by the + * above*/ + SubTransactionId create_sxid; /* subxid where this entry is created */ + SubTransactionId invalidate_sxid; /* subxid where this entry is + * invalidated */ +} RelWalRequirement; + +/* Relations that need to be fsync'd at commit */ +static HTAB *walRequirements = NULL; + +static RelWalRequirement *getWalRequirementEntry(Relation rel, bool create); +static RelWalRequirement *getWalRequirementEntryRNode(RelFileNode *node, + bool create); + /* * RelationCreateStorage * Create physical storage for a relation. @@ -259,37 +311,290 @@ RelationTruncate(Relation rel, BlockNumber nblocks) */ if (RelationNeedsWAL(rel)) { - /* - * Make an XLOG entry reporting the file truncation. - */ - XLogRecPtr lsn; - xl_smgr_truncate xlrec; + RelWalRequirement *walreq; - xlrec.blkno = nblocks; - xlrec.rnode = rel->rd_node; - xlrec.flags = SMGR_TRUNCATE_ALL; + /* get pending sync entry, create if not yet */ + walreq = getWalRequirementEntry(rel, true); - XLogBeginInsert(); - XLogRegisterData((char *) &xlrec, sizeof(xlrec)); + if (walreq->skip_wal_min_blk == InvalidBlockNumber || + walreq->skip_wal_min_blk < nblocks) + { + /* + * This is the first time truncation of this relation in this + * transaction or truncation that leaves pages that need at-commit + * fsync. Make an XLOG entry reporting the file truncation. + */ + XLogRecPtr lsn; + xl_smgr_truncate xlrec; - lsn = XLogInsert(RM_SMGR_ID, - XLOG_SMGR_TRUNCATE | XLR_SPECIAL_REL_UPDATE); + xlrec.blkno = nblocks; + xlrec.rnode = rel->rd_node; + xlrec.flags = SMGR_TRUNCATE_ALL; - /* - * Flush, because otherwise the truncation of the main relation might - * hit the disk before the WAL record, and the truncation of the FSM - * or visibility map. If we crashed during that window, we'd be left - * with a truncated heap, but the FSM or visibility map would still - * contain entries for the non-existent heap pages. - */ - if (fsm || vm) - XLogFlush(lsn); + XLogBeginInsert(); + XLogRegisterData((char *) &xlrec, sizeof(xlrec)); + + lsn = XLogInsert(RM_SMGR_ID, + XLOG_SMGR_TRUNCATE | XLR_SPECIAL_REL_UPDATE); + + /* + * Flush, because otherwise the truncation of the main relation + * might hit the disk before the WAL record, and the truncation of + * the FSM or visibility map. If we crashed during that window, + * we'd be left with a truncated heap, but the FSM or visibility + * map would still contain entries for the non-existent heap + * pages. + */ + if (fsm || vm) + XLogFlush(lsn); + + /* no longer skip WAL-logging for the blocks */ + rel->rd_walrequirement->wal_log_min_blk = nblocks; + } } /* Do the real work */ smgrtruncate(rel->rd_smgr, MAIN_FORKNUM, nblocks); } +/* + * Do changes to given heap page need to be WAL-logged? + * + * This takes into account any previous RecordPendingSync() requests. + * + * Note that it is required to check this before creating any WAL records for + * heap pages - it is not merely an optimization! WAL-logging a record, when + * we have already skipped a previous WAL record for the same page could lead + * to failure at WAL replay, as the "before" state expected by the record + * might not match what's on disk. Also, if the heap was truncated earlier, we + * must WAL-log any changes to the once-truncated blocks, because replaying + * the truncation record will destroy them. + */ +bool +BufferNeedsWAL(Relation rel, Buffer buf) +{ + BlockNumber blkno = InvalidBlockNumber; + RelWalRequirement *walreq; + + if (!RelationNeedsWAL(rel)) + return false; + + /* fetch existing pending sync entry */ + walreq = getWalRequirementEntry(rel, false); + + /* + * no point in doing further work if we know that we don't have special + * WAL requirement + */ + if (!walreq) + return true; + + Assert(BufferIsValid(buf)); + + blkno = BufferGetBlockNumber(buf); + + /* + * We don't skip WAL-logging for pages that once done. + */ + if (walreq->skip_wal_min_blk == InvalidBlockNumber || + walreq->skip_wal_min_blk > blkno) + return true; + + /* + * we don't skip WAL-logging for blocks that have got WAL-logged + * truncation + */ + if (walreq->wal_log_min_blk != InvalidBlockNumber && + walreq->wal_log_min_blk <= blkno) + return true; + + return false; +} + +bool +BlockNeedsWAL(Relation rel, BlockNumber blkno) +{ + RelWalRequirement *walreq; + + if (!RelationNeedsWAL(rel)) + return false; + + /* fetch exising pending sync entry */ + walreq = getWalRequirementEntry(rel, false); + + /* + * no point in doing further work if we know that we don't have special + * WAL requirement + */ + if (!walreq) + return true; + + /* + * We don't skip WAL-logging for pages that once done. + */ + if (walreq->skip_wal_min_blk == InvalidBlockNumber || + walreq->skip_wal_min_blk > blkno) + return true; + + /* + * we don't skip WAL-logging for blocks that have got WAL-logged + * truncation + */ + if (walreq->wal_log_min_blk != InvalidBlockNumber && + walreq->wal_log_min_blk <= blkno) + return true; + + return false; +} + +/* + * Remember that the given relation doesn't need WAL-logging for the blocks + * after the current block size and fo the blocks that are going to be synced + * at commit. + */ +void +RecordWALSkipping(Relation rel) +{ + RelWalRequirement *walreq; + + Assert(RelationNeedsWAL(rel)); + + /* get pending sync entry, create if not yet */ + walreq = getWalRequirementEntry(rel, true); + + /* + * Record only the first registration. + */ + if (walreq->skip_wal_min_blk != InvalidBlockNumber) + return; + + walreq->skip_wal_min_blk = RelationGetNumberOfBlocks(rel); +} + +/* + * Record commit-time file sync. This shouldn't be used mixing with + * RecordWALSkipping. + */ +void +RecordPendingSync(SMgrRelation rel, ForkNumber forknum) +{ + RelWalRequirement *walreq; + + walreq = getWalRequirementEntryRNode(&rel->smgr_rnode.node, true); + walreq->forks[forknum] = true; + walreq->skip_wal_min_blk = 0; +} + +/* + * RelationInvalidateWALRequirements() -- invalidate wal requirement entry + */ +void +RelationInvalidateWALRequirements(Relation rel) +{ + RelWalRequirement *walreq; + + /* we know we don't have one */ + if (rel->rd_nowalrequirement) + return; + + walreq = getWalRequirementEntry(rel, false); + + if (!walreq) + return; + + /* + * The state is reset at subtransaction commit/abort. No invalidation + * request must not come for the same relation in the same subtransaction. + */ + Assert(walreq->invalidate_sxid == InvalidSubTransactionId); + + walreq->invalidate_sxid = GetCurrentSubTransactionId(); +} + +/* + * getWalRequirementEntry: get WAL requirement entry. + * + * Returns WAL requirement entry for the relation. The entry tracks + * WAL-skipping blocks for the relation. The WAL-skipped blocks need fsync at + * commit time. Creates one if needed when create is true. + */ +static RelWalRequirement * +getWalRequirementEntry(Relation rel, bool create) +{ + RelWalRequirement *walreq_entry = NULL; + + if (rel->rd_walrequirement) + return rel->rd_walrequirement; + + /* we know we don't have pending sync entry */ + if (!create && rel->rd_nowalrequirement) + return NULL; + + walreq_entry = getWalRequirementEntryRNode(&rel->rd_node, create); + + if (!walreq_entry) + { + /* prevent further hash lookup */ + rel->rd_nowalrequirement = true; + return NULL; + } + + walreq_entry->forks[MAIN_FORKNUM] = true; + + /* hold shortcut in Relation */ + rel->rd_nowalrequirement = false; + rel->rd_walrequirement = walreq_entry; + + return walreq_entry; +} + +/* + * getWalRequirementEntryRNode: get WAL requirement entry by rnode + * + * Returns WAL requirement entry for the RelFileNode. + */ +static RelWalRequirement * +getWalRequirementEntryRNode(RelFileNode *rnode, bool create) +{ + RelWalRequirement *walreq_entry = NULL; + bool found; + + if (!walRequirements) + { + /* First time through: initialize the hash table */ + HASHCTL ctl; + + if (!create) + return NULL; + + MemSet(&ctl, 0, sizeof(ctl)); + ctl.keysize = sizeof(RelFileNode); + ctl.entrysize = sizeof(RelWalRequirement); + ctl.hash = tag_hash; + walRequirements = hash_create("pending relation sync table", 5, + &ctl, HASH_ELEM | HASH_FUNCTION); + } + + walreq_entry = (RelWalRequirement *) + hash_search(walRequirements, (void *) rnode, + create ? HASH_ENTER: HASH_FIND, &found); + + if (!walreq_entry) + return NULL; + + /* new entry created */ + if (!found) + { + memset(&walreq_entry->forks, 0, sizeof(sizeof(walreq_entry->forks))); + walreq_entry->wal_log_min_blk = InvalidBlockNumber; + walreq_entry->skip_wal_min_blk = InvalidBlockNumber; + walreq_entry->create_sxid = GetCurrentSubTransactionId(); + walreq_entry->invalidate_sxid = InvalidSubTransactionId; + } + + return walreq_entry; +} + /* * smgrDoPendingDeletes() -- Take care of relation deletes at end of xact. * @@ -418,6 +723,75 @@ smgrGetPendingDeletes(bool forCommit, RelFileNode **ptr) return nrels; } +/* + * Sync to disk any relations that we have skipped WAL-logging earlier. + */ +void +smgrDoPendingSyncs(bool isCommit) +{ + if (!walRequirements) + return; + + if (isCommit) + { + HASH_SEQ_STATUS status; + RelWalRequirement *walreq; + + hash_seq_init(&status, walRequirements); + + while ((walreq = hash_seq_search(&status)) != NULL) + { + if (walreq->skip_wal_min_blk != InvalidBlockNumber && + walreq->invalidate_sxid == InvalidSubTransactionId) + { + int f; + + FlushRelationBuffersWithoutRelCache(walreq->relnode, false); + + /* flush all requested forks */ + for (f = MAIN_FORKNUM ; f <= MAX_FORKNUM ; f++) + { + if (walreq->forks[f]) + smgrimmedsync(smgropen(walreq->relnode, + InvalidBackendId), f); + } + } + } + } + + hash_destroy(walRequirements); + walRequirements = NULL; +} + +/* + * Process pending invalidation of WAL requirements happened in the + * subtransaction + */ +void +smgrProcessWALRequirementInval(SubTransactionId sxid, bool isCommit) +{ + HASH_SEQ_STATUS status; + RelWalRequirement *walreq; + + if (!walRequirements) + return; + + /* We expect that we don't have walRequirements in almost all cases */ + hash_seq_init(&status, walRequirements); + + while ((walreq = hash_seq_search(&status)) != NULL) + { + /* remove useless entry */ + if (isCommit ? + walreq->invalidate_sxid == sxid : + walreq->create_sxid == sxid) + hash_search(walRequirements, &walreq->relnode, HASH_REMOVE, NULL); + /* or cancel invalidation */ + else if (!isCommit && walreq->invalidate_sxid == sxid) + walreq->invalidate_sxid = InvalidSubTransactionId; + } +} + /* * PostPrepare_smgr -- Clean up after a successful PREPARE * diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 3183b2aaa1..c9a0e02168 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11587,11 +11587,13 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode) /* * Create and copy all forks of the relation, and schedule unlinking of - * old physical files. + * old physical files. WAL requirements for the old node is no longer + * needed. * * NOTE: any conflict in relfilenode value will be caught in * RelationCreateStorage(). */ + RelationInvalidateWALRequirements(rel); RelationCreateStorage(newrnode, rel->rd_rel->relpersistence); /* copy main fork */ diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 273e2f385f..f00826712a 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -451,6 +451,7 @@ static BufferDesc *BufferAlloc(SMgrRelation smgr, BufferAccessStrategy strategy, bool *foundPtr); static void FlushBuffer(BufferDesc *buf, SMgrRelation reln); +static void FlushRelationBuffers_common(SMgrRelation smgr, bool islocal); static void AtProcExit_Buffers(int code, Datum arg); static void CheckForBufferLeaks(void); static int rnode_comparator(const void *p1, const void *p2); @@ -3153,20 +3154,40 @@ PrintPinnedBufs(void) void FlushRelationBuffers(Relation rel) { - int i; - BufferDesc *bufHdr; - /* Open rel at the smgr level if not already done */ RelationOpenSmgr(rel); - if (RelationUsesLocalBuffers(rel)) + FlushRelationBuffers_common(rel->rd_smgr, RelationUsesLocalBuffers(rel)); +} + +/* + * Like FlushRelationBuffers(), but the relation is specified by RelFileNode + */ +void +FlushRelationBuffersWithoutRelCache(RelFileNode rnode, bool islocal) +{ + FlushRelationBuffers_common(smgropen(rnode, InvalidBackendId), islocal); +} + +/* + * Code shared between functions FlushRelationBuffers() and + * FlushRelationBuffersWithoutRelCache(). + */ +static void +FlushRelationBuffers_common(SMgrRelation smgr, bool islocal) +{ + RelFileNode rnode = smgr->smgr_rnode.node; + int i; + BufferDesc *bufHdr; + + if (islocal) { for (i = 0; i < NLocBuffer; i++) { uint32 buf_state; bufHdr = GetLocalBufferDescriptor(i); - if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && + if (RelFileNodeEquals(bufHdr->tag.rnode, rnode) && ((buf_state = pg_atomic_read_u32(&bufHdr->state)) & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { @@ -3183,7 +3204,7 @@ FlushRelationBuffers(Relation rel) PageSetChecksumInplace(localpage, bufHdr->tag.blockNum); - smgrwrite(rel->rd_smgr, + smgrwrite(smgr, bufHdr->tag.forkNum, bufHdr->tag.blockNum, localpage, @@ -3213,18 +3234,18 @@ FlushRelationBuffers(Relation rel) * As in DropRelFileNodeBuffers, an unlocked precheck should be safe * and saves some cycles. */ - if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) + if (!RelFileNodeEquals(bufHdr->tag.rnode, rnode)) continue; ReservePrivateRefCountEntry(); buf_state = LockBufHdr(bufHdr); - if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && + if (RelFileNodeEquals(bufHdr->tag.rnode, rnode) && (buf_state & (BM_VALID | BM_DIRTY)) == (BM_VALID | BM_DIRTY)) { PinBuffer_Locked(bufHdr); LWLockAcquire(BufferDescriptorGetContentLock(bufHdr), LW_SHARED); - FlushBuffer(bufHdr, rel->rd_smgr); + FlushBuffer(bufHdr, smgr); LWLockRelease(BufferDescriptorGetContentLock(bufHdr)); UnpinBuffer(bufHdr, true); } diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c index 84609e0725..95e834d45e 100644 --- a/src/backend/utils/cache/relcache.c +++ b/src/backend/utils/cache/relcache.c @@ -75,6 +75,7 @@ #include "partitioning/partdesc.h" #include "rewrite/rewriteDefine.h" #include "rewrite/rowsecurity.h" +#include "storage/bufmgr.h" #include "storage/lmgr.h" #include "storage/smgr.h" #include "utils/array.h" @@ -5625,6 +5626,8 @@ load_relcache_init_file(bool shared) rel->rd_newRelfilenodeSubid = InvalidSubTransactionId; rel->rd_amcache = NULL; MemSet(&rel->pgstat_info, 0, sizeof(rel->pgstat_info)); + rel->rd_nowalrequirement = false; + rel->rd_walrequirement = NULL; /* * Recompute lock and physical addressing info. This is needed in diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index 3773a4df85..3d4fb7f3c3 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -172,6 +172,7 @@ extern void simple_heap_delete(Relation relation, ItemPointer tid); extern void simple_heap_update(Relation relation, ItemPointer otid, HeapTuple tup); +extern void heap_register_sync(Relation relation); extern void heap_sync(Relation relation); /* in heap/pruneheap.c */ diff --git a/src/include/catalog/storage.h b/src/include/catalog/storage.h index 9f638be924..9034465001 100644 --- a/src/include/catalog/storage.h +++ b/src/include/catalog/storage.h @@ -16,12 +16,18 @@ #include "storage/block.h" #include "storage/relfilenode.h" +#include "storage/smgr.h" #include "utils/relcache.h" extern void RelationCreateStorage(RelFileNode rnode, char relpersistence); extern void RelationDropStorage(Relation rel); extern void RelationPreserveStorage(RelFileNode rnode, bool atCommit); extern void RelationTruncate(Relation rel, BlockNumber nblocks); +extern bool BufferNeedsWAL(Relation rel, Buffer buf); +extern bool BlockNeedsWAL(Relation rel, BlockNumber blkno); +extern void RecordWALSkipping(Relation rel); +extern void RecordPendingSync(SMgrRelation rel, ForkNumber forknum); +extern void RelationInvalidateWALRequirements(Relation rel); /* * These functions used to be in storage/smgr/smgr.c, which explains the @@ -29,6 +35,8 @@ extern void RelationTruncate(Relation rel, BlockNumber nblocks); */ extern void smgrDoPendingDeletes(bool isCommit); extern int smgrGetPendingDeletes(bool forCommit, RelFileNode **ptr); +extern void smgrDoPendingSyncs(bool isCommit); +extern void smgrProcessWALRequirementInval(SubTransactionId sxid, bool isCommit); extern void AtSubCommit_smgr(void); extern void AtSubAbort_smgr(void); extern void PostPrepare_smgr(void); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index c5826f691d..8a9ea041dd 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -189,6 +189,8 @@ extern BlockNumber RelationGetNumberOfBlocksInFork(Relation relation, ForkNumber forkNum); extern void FlushOneBuffer(Buffer buffer); extern void FlushRelationBuffers(Relation rel); +extern void FlushRelationBuffersWithoutRelCache(RelFileNode rnode, + bool islocal); extern void FlushDatabaseBuffers(Oid dbid); extern void DropRelFileNodeBuffers(RelFileNodeBackend rnode, ForkNumber forkNum, BlockNumber firstDelBlock); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 54028515a7..30f0d5bd83 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -198,6 +198,14 @@ typedef struct RelationData /* use "struct" here to avoid needing to include pgstat.h: */ struct PgStat_TableStatus *pgstat_info; /* statistics collection area */ + + /* + * rd_nowalrequirement is true if this relation is known not to have + * special WAL requirements. Otherwise we need to ask smgr for an entry + * if rd_walrequirement is NULL. + */ + bool rd_nowalrequirement; + struct RelWalRequirement *rd_walrequirement; } RelationData; -- 2.16.3 ----Next_Part(Tue_Mar_26_16_35_07_2019_128)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v9-0005-Fix-WAL-skipping-feature.patch" ^ permalink raw reply [nested|flat] 3+ messages in thread
* [PATCH v26 3/9] Row pattern recognition patch (rewriter). @ 2024-12-30 12:44 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Tatsuo Ishii @ 2024-12-30 12:44 UTC (permalink / raw) --- src/backend/utils/adt/ruleutils.c | 103 ++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index be1f1f50b7..f58392f0aa 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -435,6 +435,10 @@ static void get_rule_groupingset(GroupingSet *gset, List *targetlist, bool omit_parens, deparse_context *context); static void get_rule_orderby(List *orderList, List *targetList, bool force_colno, deparse_context *context); +static void get_rule_pattern(List *patternVariable, List *patternRegexp, + bool force_colno, deparse_context *context); +static void get_rule_define(List *defineClause, List *patternVariables, + bool force_colno, deparse_context *context); static void get_rule_windowclause(Query *query, deparse_context *context); static void get_rule_windowspec(WindowClause *wc, List *targetList, deparse_context *context); @@ -6667,6 +6671,67 @@ get_rule_orderby(List *orderList, List *targetList, } } +/* + * Display a PATTERN clause. + */ +static void +get_rule_pattern(List *patternVariable, List *patternRegexp, + bool force_colno, deparse_context *context) +{ + StringInfo buf = context->buf; + const char *sep; + ListCell *lc_var, + *lc_reg = list_head(patternRegexp); + + sep = ""; + appendStringInfoChar(buf, '('); + foreach(lc_var, patternVariable) + { + char *variable = strVal((String *) lfirst(lc_var)); + char *regexp = NULL; + + if (lc_reg != NULL) + { + regexp = strVal((String *) lfirst(lc_reg)); + + lc_reg = lnext(patternRegexp, lc_reg); + } + + appendStringInfo(buf, "%s%s", sep, variable); + if (regexp !=NULL) + appendStringInfoString(buf, regexp); + + sep = " "; + } + appendStringInfoChar(buf, ')'); +} + +/* + * Display a DEFINE clause. + */ +static void +get_rule_define(List *defineClause, List *patternVariables, + bool force_colno, deparse_context *context) +{ + StringInfo buf = context->buf; + const char *sep; + ListCell *lc_var, + *lc_def; + + sep = " "; + Assert(list_length(patternVariables) == list_length(defineClause)); + + forboth(lc_var, patternVariables, lc_def, defineClause) + { + char *varName = strVal(lfirst(lc_var)); + TargetEntry *te = (TargetEntry *) lfirst(lc_def); + + appendStringInfo(buf, "%s%s AS ", sep, varName); + get_rule_expr((Node *) te->expr, context, false); + sep = ",\n "; + } +} + /* * Display a WINDOW clause. * @@ -6804,6 +6869,44 @@ get_rule_windowspec(WindowClause *wc, List *targetList, appendStringInfoString(buf, "EXCLUDE GROUP "); else if (wc->frameOptions & FRAMEOPTION_EXCLUDE_TIES) appendStringInfoString(buf, "EXCLUDE TIES "); + /* RPR */ + if (wc->rpSkipTo == ST_NEXT_ROW) + appendStringInfoString(buf, + "\n AFTER MATCH SKIP TO NEXT ROW "); + else if (wc->rpSkipTo == ST_PAST_LAST_ROW) + appendStringInfoString(buf, + "\n AFTER MATCH SKIP PAST LAST ROW "); + else if (wc->rpSkipTo == ST_FIRST_VARIABLE) + appendStringInfo(buf, + "\n AFTER MATCH SKIP TO FIRST %s ", + wc->rpSkipVariable); + else if (wc->rpSkipTo == ST_LAST_VARIABLE) + appendStringInfo(buf, + "\n AFTER MATCH SKIP TO LAST %s ", + wc->rpSkipVariable); + else if (wc->rpSkipTo == ST_VARIABLE) + appendStringInfo(buf, + "\n AFTER MATCH SKIP TO %s ", + wc->rpSkipVariable); + + if (wc->initial) + appendStringInfoString(buf, "\n INITIAL"); + + if (wc->patternVariable) + { + appendStringInfoString(buf, "\n PATTERN "); + get_rule_pattern(wc->patternVariable, wc->patternRegexp, + false, context); + } + + if (wc->defineClause) + { + appendStringInfoString(buf, "\n DEFINE\n"); + get_rule_define(wc->defineClause, wc->patternVariable, + false, context); + appendStringInfoChar(buf, ' '); + } + /* we will now have a trailing space; remove it */ buf->len--; } -- 2.25.1 ----Next_Part(Mon_Dec_30_22_37_18_2024_171)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v26-0004-Row-pattern-recognition-patch-planner.patch" ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Optimize scram_SaltedPassword performance @ 2025-02-03 11:45 Yura Sokolov <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Yura Sokolov @ 2025-02-03 11:45 UTC (permalink / raw) To: Daniel Gustafsson <[email protected]>; Zixuan Fu <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]> 03.02.2025 14:17, Daniel Gustafsson пишет: >> On 3 Feb 2025, at 08:45, Yura Sokolov <[email protected]> wrote: >> >> 03.02.2025 10:11, Zixuan Fu пишет: >>> Hi Hackers, >>> >>> While profiling a program with `perf`, I noticed that `scram_SaltedPassword` >>> consumed more CPU time than expected. After some investigation, I found >>> that the function performs many HMAC iterations (4096 rounds for >>> SCRAM-SHA-256), and each iteration reinitializes the HMAC context, causing >>> excessive overhead. > > While I don't disagree with speeding up this in general, the whole point of the > SCRAM iterations is to take a lot of time as a way to combat brute forcing. > Any attacker is likely to use a patched libpq (or not use libpq at all) so > clientside it doesn't matter as much but if we make it 4x faster serverside we > don't really achieve much other than making attacks more feasible. > > The relevant portion from RFC 7677 §4 is: > > The strength of this mechanism is dependent in part on the hash > iteration-count, as denoted by "i" in [RFC5802]. As a rule of thumb, > the hash iteration-count should be such that a modern machine will take > 0.1 seconds to perform the complete algorithm; however, this is > unlikely to be practical on mobile devices and other relatively low- > performance systems. At the time this was written, the rule of thumb > gives around 15,000 iterations required; however, a hash iteration- > count of 4096 takes around 0.5 seconds on current mobile handsets. > This computational cost can be avoided by caching the ClientKey > (assuming the Salt and hash iteration-count is stable). Therefore, the > recommendation of this specification is that the hash iteration- count > SHOULD be at least 4096, but careful consideration ought to be given to > using a significantly higher value, particularly where mobile use is > less important. > > The numbers are quite outdated but the gist of it holds. If we speed it up > serverside we need to counter that with a higher iteration count, and for a > rogue client it's unlikely to matter since it wont use our code anyways. > Speeding up HMAC for other usecases is a different story (but also likely to > have less measurable performance impact). There is no sense to not speedup server if client can be made faster. HMAC should protect from an attacker who have very optimized software on very powerful hardware, since it should protect against BRUTE force against known cipher text either. If 4096 iterations could be performed fast in attacker's environment, there's no point to slow down our environment. It is meaningless. >>> OpenSSL has an optimization for this case: when the key remains the >>> same, the HMAC context can be reused with a lightweight state reset by >>> passing NULL as the key. To take advantage of this, I introduced >>> `pg_hmac_reuse()`, which replaces the key with NULL when OpenSSL is used. > >> Where `prev_key`, `prev_len` and `prev_hash` are static variables, filled >> in `pg_hmac_init`. > > Storing any part of a cryptograhic calculation, let alone a key, in a static > variable doesn't seem entirely like a best practice, and it also wont be > threadsafe. `prev_key` stores not the key, but pointer to key. Just to ensure `pg_hmac_reuse` were called with exactly same key. It doesn't leak the key itself. Storing `prev_hash` is more considered as "storing part of key". I may agree it is not exactly good idea if `key` is "human-brain-generated". Thread-safety is more-or-less trivial with `thread_local` and synonyms for. -- Yura ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2025-02-03 11:45 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2019-03-26 06:34 [PATCH 4/8] Add infrastructure to WAL-logging skip feature Kyotaro Horiguchi <[email protected]> 2024-12-30 12:44 [PATCH v26 3/9] Row pattern recognition patch (rewriter). Tatsuo Ishii <[email protected]> 2025-02-03 11:45 Re: Optimize scram_SaltedPassword performance Yura Sokolov <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox