public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. 92+ messages / 3 participants [nested] [flat]
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v10 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index e88e4e918b0..cde9184c7d6 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1174,9 +1171,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1677,8 +1676,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1689,6 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1769,7 +1768,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1945,9 +1945,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2421,9 +2418,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2505,8 +2499,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2516,7 +2508,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3555,9 +3549,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3571,7 +3562,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3628,9 +3621,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3667,7 +3657,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3707,9 +3699,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3723,7 +3712,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 4fbdc62d8c7..13eed587601 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.2 --------------AF226508B1039DE40796DE51 Content-Type: text/x-patch; charset=UTF-8; name="v10-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v10-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------F30A5443A5D524A5CBC51AE7 Content-Type: text/x-patch; charset=UTF-8; name="v6-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v6-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------1104342304EC6341F5D7C60C Content-Type: text/x-patch; charset=UTF-8; name="v5-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v5-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v8 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 86ef607ff38..2383e4cac08 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1165,9 +1162,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1668,8 +1667,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1680,6 +1677,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1760,7 +1759,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1936,9 +1936,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2412,9 +2409,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2496,8 +2490,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2507,7 +2499,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3559,9 +3553,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3575,7 +3566,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3632,9 +3625,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3671,7 +3661,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3711,9 +3703,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3727,7 +3716,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 4fbdc62d8c7..13eed587601 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.2 --------------846F221A8A016B7AA8E567B3 Content-Type: text/x-patch; charset=UTF-8; name="v8-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v8-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index 55c94458981..2630c800b05 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.30.1 --------------C6A03312D0F99F9183A0DF46 Content-Type: text/x-patch; charset=UTF-8; name="v7-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v7-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. @ 2021-01-13 10:21 Heikki Linnakangas <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw) These are functions where quite a lot of things happen between the ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that there are no unrelated ResourceOwnerRemember() calls in the code inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call might be used up by the intervening ResourceOwnerRemember() and not be available at the intended ResourceOwnerRemember() call anymore. The longer the code path between them is, the harder it is to verify that. In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to ensure that the private refcount array has enough space. The ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(), were made at different places than the ResourceOwnerEnlarge() calls. Move the ResourceOwnerEnlarge() calls together with the ReservePrivateRefCountEntry() calls for consistency. --- src/backend/storage/buffer/bufmgr.c | 39 +++++++++++---------------- src/backend/storage/buffer/localbuf.c | 3 +++ src/backend/utils/cache/catcache.c | 13 ++++++--- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 561c212092f..cde869e7d64 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, *hit = false; - /* Make sure we will have room to remember the buffer pin */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - isExtend = (blockNum == P_NEW); TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum, @@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, { /* * Ensure, while the spinlock's not yet held, that there's a free - * refcount entry. + * refcount entry and that the resoure owner has room to remember the + * pin. */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Select a victim buffer. The buffer is returned with its header @@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer, * taking the buffer header lock; instead update the state variable in loop of * CAS operations. Hopefully it's just a single CAS. * - * Note that ResourceOwnerEnlargeBuffers must have been done already. - * * Returns true if buffer is BM_VALID, else false. This provision allows * some callers to avoid an extra spinlock cycle. */ @@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) bool result; PrivateRefCountEntry *ref; + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + ref = GetPrivateRefCountEntry(b, true); if (ref == NULL) @@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy) * The spinlock is released before return. * * As this function is called with the spinlock held, the caller has to - * previously call ReservePrivateRefCountEntry(). + * previously call ReservePrivateRefCountEntry() and + * ResourceOwnerEnlargeBuffers(CurrentResourceOwner); * * Currently, no callers of this function want to modify the buffer's * usage_count at all, so there's no need for a strategy parameter. @@ -1857,9 +1857,6 @@ BufferSync(int flags) int mask = BM_DIRTY; WritebackContext wb_context; - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - /* * Unless this is a shutdown checkpoint or we have been explicitly told, * we write only permanent, dirty buffers. But at shutdown or end of @@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context) * requirements, or hit the bgwriter_lru_maxpages limit. */ - /* Make sure we can handle the pin inside SyncOneBuffer */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - num_to_scan = bufs_to_lap; num_written = 0; reusable_buffers = reusable_buffers_est; @@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context) * * (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean * after locking it, but we don't care all that much.) - * - * Note: caller must have done ResourceOwnerEnlargeBuffers. */ static int SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) @@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context) uint32 buf_state; BufferTag tag; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); /* * Check whether buffer needs writing. @@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel) return; } - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel) if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node)) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) && @@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (use_bsearch) pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator); - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { SMgrSortArray *srelent = NULL; @@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels) if (srelent == NULL) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) && @@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid) int i; BufferDesc *bufHdr; - /* Make sure we can handle the pin inside the loop */ - ResourceOwnerEnlargeBuffers(CurrentResourceOwner); - for (i = 0; i < NBuffers; i++) { uint32 buf_state; @@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid) if (bufHdr->tag.rnode.dbNode != dbid) continue; + /* Make sure we can handle the pin */ ReservePrivateRefCountEntry(); + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); buf_state = LockBufHdr(bufHdr); if (bufHdr->tag.rnode.dbNode == dbid && diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c index 04b3558ea33..f7c15ea8a44 100644 --- a/src/backend/storage/buffer/localbuf.c +++ b/src/backend/storage/buffer/localbuf.c @@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum, if (LocalBufHash == NULL) InitLocalBuffers(); + /* Make sure we will have room to remember the buffer pin */ + ResourceOwnerEnlargeBuffers(CurrentResourceOwner); + /* See if the desired buffer already exists */ hresult = (LocalBufferLookupEnt *) hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL); diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c index fa2b49c676e..86fdfc454d2 100644 --- a/src/backend/utils/cache/catcache.c +++ b/src/backend/utils/cache/catcache.c @@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache, * block to ensure we can undo those refcounts if we get an error before * we finish constructing the CatCList. */ - ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); - ctlist = NIL; PG_TRY(); @@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache, table_close(relation, AccessShareLock); + /* Make sure the resource owner has room to remember this entry. */ + ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner); + /* Now we can build the CatCList entry. */ oldcxt = MemoryContextSwitchTo(CacheMemoryContext); nmembers = list_length(ctlist); cl = (CatCList *) palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *)); - /* Extract key values */ + /* + * Extract key values. + * + * XXX: If we run out of memory while copying the key values, we will + * leak any allocations we had already made in the CacheMemoryContext. + * That is unlikely enough that we just accept the risk. + */ CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno, arguments, cl->keys); MemoryContextSwitchTo(oldcxt); -- 2.29.2 --------------7349B9D66ED595EF3B1030B4 Content-Type: text/x-patch; charset=UTF-8; name="v4-0002-Make-resowners-more-easily-extensible.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="v4-0002-Make-resowners-more-easily-extensible.patch" ^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: row filtering for logical replication @ 2022-04-12 09:05 Alvaro Herrera <[email protected]> 2022-04-12 10:15 ` Re: row filtering for logical replication Amit Kapila <[email protected]> 0 siblings, 1 reply; 92+ messages in thread From: Alvaro Herrera @ 2022-04-12 09:05 UTC (permalink / raw) To: [email protected] <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Tomas Vondra <[email protected]>; Euler Taveira <[email protected]>; [email protected] <[email protected]>; Greg Nancarrow <[email protected]>; vignesh C <[email protected]>; Ajin Cherian <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Rahila Syed <[email protected]>; Peter Eisentraut <[email protected]>; Önder Kalacı <[email protected]>; japin <[email protected]>; Michael Paquier <[email protected]>; David Steele <[email protected]>; Craig Ringer <[email protected]>; Amit Langote <[email protected]>; PostgreSQL Hackers <[email protected]> I understand that this is a minimal fix, and for that it seems OK, but I think the surrounding style is rather baroque. This code can be made simpler. Here's my take on it. I think it's also faster: we avoid looking up pg_publication_rel entries for rels that aren't partitioned tables. -- Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/ "Cuando mañana llegue pelearemos segun lo que mañana exija" (Mowgli) Attachments: [text/x-diff] style.patch (4.8K, ../../[email protected]/2-style.patch) download | inline diff: diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c index 7aacb6b2fe..e8ef003fe5 100644 --- a/src/backend/commands/publicationcmds.c +++ b/src/backend/commands/publicationcmds.c @@ -945,60 +945,42 @@ AlterPublicationOptions(ParseState *pstate, AlterPublicationStmt *stmt, foreach(lc, root_relids) { - HeapTuple rftuple; Oid relid = lfirst_oid(lc); - bool has_column_list; - bool has_row_filter; + char relkind; + HeapTuple rftuple; + + relkind = get_rel_relkind(relid); + if (relkind != RELKIND_PARTITIONED_TABLE) + continue; rftuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(relid), ObjectIdGetDatum(pubform->oid)); + if (!HeapTupleIsValid(rftuple)) + continue; - has_row_filter - = !heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL); + if (!heap_attisnull(rftuple, Anum_pg_publication_rel_prqual, NULL)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("cannot set %s to false for publication \"%s\"", + "publish_via_partition_root", + stmt->pubname), + errdetail("The publication contains a WHERE clause for a partitioned table \"%s\" which is not allowed when %s is false.", + get_rel_name(relid), + "publish_via_partition_root"))); - has_column_list - = !heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL); + if (!heap_attisnull(rftuple, Anum_pg_publication_rel_prattrs, NULL)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("cannot set %s to false for publication \"%s\"", + "publish_via_partition_root", + stmt->pubname), + errdetail("The publication contains a column list for a partitioned table \"%s\" which is not allowed when %s is false.", + get_rel_name(relid), + "publish_via_partition_root"))); - if (HeapTupleIsValid(rftuple) && - (has_row_filter || has_column_list)) - { - HeapTuple tuple; + ReleaseSysCache(rftuple); - tuple = SearchSysCache1(RELOID, ObjectIdGetDatum(relid)); - if (HeapTupleIsValid(tuple)) - { - Form_pg_class relform = (Form_pg_class) GETSTRUCT(tuple); - - if ((relform->relkind == RELKIND_PARTITIONED_TABLE) && - has_row_filter) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("cannot set %s for publication \"%s\"", - "publish_via_partition_root = false", - stmt->pubname), - errdetail("The publication contains a WHERE clause for a partitioned table \"%s\" " - "which is not allowed when %s is false.", - NameStr(relform->relname), - "publish_via_partition_root"))); - - if ((relform->relkind == RELKIND_PARTITIONED_TABLE) && - has_column_list) - ereport(ERROR, - (errcode(ERRCODE_INVALID_PARAMETER_VALUE), - errmsg("cannot set %s for publication \"%s\"", - "publish_via_partition_root = false", - stmt->pubname), - errdetail("The publication contains a column list for a partitioned table \"%s\" " - "which is not allowed when %s is false.", - NameStr(relform->relname), - "publish_via_partition_root"))); - - ReleaseSysCache(tuple); - } - - ReleaseSysCache(rftuple); - } } } diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out index 8208f9fa0e..580cc5be7f 100644 --- a/src/test/regress/expected/publication.out +++ b/src/test/regress/expected/publication.out @@ -588,7 +588,7 @@ UPDATE rf_tbl_abcd_part_pk SET a = 1; -- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any row filter is -- used for partitioned table ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0); -ERROR: cannot set publish_via_partition_root = false for publication "testpub6" +ERROR: cannot set publish_via_partition_root to false for publication "testpub6" DETAIL: The publication contains a WHERE clause for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false. -- Now change the root filter to use a column "b" -- (which is not in the replica identity) @@ -951,7 +951,7 @@ UPDATE rf_tbl_abcd_part_pk SET a = 1; -- fail - cannot set PUBLISH_VIA_PARTITION_ROOT to false if any column list is -- used for partitioned table ALTER PUBLICATION testpub6 SET (PUBLISH_VIA_PARTITION_ROOT=0); -ERROR: cannot set publish_via_partition_root = false for publication "testpub6" +ERROR: cannot set publish_via_partition_root to false for publication "testpub6" DETAIL: The publication contains a column list for a partitioned table "rf_tbl_abcd_part_pk" which is not allowed when publish_via_partition_root is false. -- Now change the root column list to use a column "b" -- (which is not in the replica identity) ^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: row filtering for logical replication 2022-04-12 09:05 Re: row filtering for logical replication Alvaro Herrera <[email protected]> @ 2022-04-12 10:15 ` Amit Kapila <[email protected]> 2022-04-12 10:30 ` Re: row filtering for logical replication Amit Kapila <[email protected]> 0 siblings, 1 reply; 92+ messages in thread From: Amit Kapila @ 2022-04-12 10:15 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; Tomas Vondra <[email protected]>; Euler Taveira <[email protected]>; [email protected] <[email protected]>; Greg Nancarrow <[email protected]>; vignesh C <[email protected]>; Ajin Cherian <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Rahila Syed <[email protected]>; Peter Eisentraut <[email protected]>; Önder Kalacı <[email protected]>; japin <[email protected]>; Michael Paquier <[email protected]>; David Steele <[email protected]>; Craig Ringer <[email protected]>; Amit Langote <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Apr 12, 2022 at 2:35 PM Alvaro Herrera <[email protected]> wrote: > > I understand that this is a minimal fix, and for that it seems OK, but I > think the surrounding style is rather baroque. This code can be made > simpler. Here's my take on it. > We don't have a lock on the relation, so if it gets dropped concurrently, it won't behave sanely. For example, get_rel_name() will return NULL which seems incorrect to me. > I think it's also faster: we avoid > looking up pg_publication_rel entries for rels that aren't partitioned > tables. > I am not sure about this as well because you will instead do a RELOID cache lookup even when there is no row filter or column list. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: row filtering for logical replication 2022-04-12 09:05 Re: row filtering for logical replication Alvaro Herrera <[email protected]> 2022-04-12 10:15 ` Re: row filtering for logical replication Amit Kapila <[email protected]> @ 2022-04-12 10:30 ` Amit Kapila <[email protected]> 0 siblings, 0 replies; 92+ messages in thread From: Amit Kapila @ 2022-04-12 10:30 UTC (permalink / raw) To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; Peter Smith <[email protected]>; Tomas Vondra <[email protected]>; Euler Taveira <[email protected]>; [email protected] <[email protected]>; Greg Nancarrow <[email protected]>; vignesh C <[email protected]>; Ajin Cherian <[email protected]>; [email protected] <[email protected]>; Dilip Kumar <[email protected]>; Rahila Syed <[email protected]>; Peter Eisentraut <[email protected]>; Önder Kalacı <[email protected]>; japin <[email protected]>; Michael Paquier <[email protected]>; David Steele <[email protected]>; Craig Ringer <[email protected]>; Amit Langote <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Apr 12, 2022 at 3:45 PM Amit Kapila <[email protected]> wrote: > > On Tue, Apr 12, 2022 at 2:35 PM Alvaro Herrera <[email protected]> wrote: > > > > I understand that this is a minimal fix, and for that it seems OK, but I > > think the surrounding style is rather baroque. This code can be made > > simpler. Here's my take on it. > > > > We don't have a lock on the relation, so if it gets dropped > concurrently, it won't behave sanely. For example, get_rel_name() will > return NULL which seems incorrect to me. > It seems to me that we have a similar coding pattern in ExecGrant_Relation(). -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 92+ messages in thread
end of thread, other threads:[~2022-04-12 10:30 UTC | newest] Thread overview: 92+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v10 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v8 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]> 2022-04-12 09:05 Re: row filtering for logical replication Alvaro Herrera <[email protected]> 2022-04-12 10:15 ` Re: row filtering for logical replication Amit Kapila <[email protected]> 2022-04-12 10:30 ` Re: row filtering for logical replication Amit Kapila <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox