public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
92+ messages / 3 participants
[nested] [flat]
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v10 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e88e4e918b0..cde9184c7d6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1174,9 +1171,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1677,8 +1676,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1689,6 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1769,7 +1768,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1945,9 +1945,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2421,9 +2418,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2505,8 +2499,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2516,7 +2508,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3555,9 +3549,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3571,7 +3562,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3628,9 +3621,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3667,7 +3657,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3707,9 +3699,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3723,7 +3712,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 4fbdc62d8c7..13eed587601 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.2
--------------AF226508B1039DE40796DE51
Content-Type: text/x-patch; charset=UTF-8;
name="v10-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v10-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v8 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 86ef607ff38..2383e4cac08 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1165,9 +1162,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1668,8 +1667,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1680,6 +1677,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1760,7 +1759,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1936,9 +1936,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2412,9 +2409,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2496,8 +2490,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2507,7 +2499,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3559,9 +3553,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3575,7 +3566,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3632,9 +3625,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3671,7 +3661,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3711,9 +3703,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3727,7 +3716,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 4fbdc62d8c7..13eed587601 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.2
--------------846F221A8A016B7AA8E567B3
Content-Type: text/x-patch; charset=UTF-8;
name="v8-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v8-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------1104342304EC6341F5D7C60C
Content-Type: text/x-patch; charset=UTF-8;
name="v5-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v5-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------7349B9D66ED595EF3B1030B4
Content-Type: text/x-patch; charset=UTF-8;
name="v4-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v4-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 55c94458981..2630c800b05 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.1
--------------C6A03312D0F99F9183A0DF46
Content-Type: text/x-patch; charset=UTF-8;
name="v7-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v7-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity.
@ 2021-01-13 10:21 Heikki Linnakangas <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 10:21 UTC (permalink / raw)
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 561c212092f..cde869e7d64 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -738,9 +738,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1093,9 +1090,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1595,8 +1594,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1607,6 +1604,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1687,7 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1857,9 +1857,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2334,9 +2331,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2418,8 +2412,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2429,7 +2421,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3487,9 +3481,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3503,7 +3494,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3560,9 +3553,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3599,7 +3589,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3639,9 +3631,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3655,7 +3644,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index fa2b49c676e..86fdfc454d2 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1607,8 +1607,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1696,13 +1694,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.29.2
--------------F30A5443A5D524A5CBC51AE7
Content-Type: text/x-patch; charset=UTF-8;
name="v6-0002-Make-resowners-more-easily-extensible.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="v6-0002-Make-resowners-more-easily-extensible.patch"
^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: cataloguing NOT NULL constraints
@ 2024-04-18 13:39 Alvaro Herrera <[email protected]>
2024-04-18 20:00 ` Re: cataloguing NOT NULL constraints Alexander Lakhin <[email protected]>
0 siblings, 1 reply; 92+ messages in thread
From: Alvaro Herrera @ 2024-04-18 13:39 UTC (permalink / raw)
To: Andrew Bille <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Peter Eisentraut <[email protected]>; Robert Haas <[email protected]>; [email protected]
On 2024-Jan-25, Andrew Bille wrote:
> Starting from b0e96f31, pg_upgrade fails with inherited NOT NULL constraint:
> For example upgrade from 9c13b6814a (or REL_12_STABLE .. REL_16_STABLE) to
> b0e96f31 (or master) with following two tables (excerpt from
> src/test/regress/sql/rules.sql)
>
> create table test_0 (id serial primary key);
> create table test_1 (id integer primary key) inherits (test_0);
I have pushed a fix which should hopefully fix this problem
(d9f686a72e). Please give this a look. Thanks for reporting the issue.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"I apologize for the confusion in my previous responses.
There appears to be an error." (ChatGPT)
^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: cataloguing NOT NULL constraints
2024-04-18 13:39 Re: cataloguing NOT NULL constraints Alvaro Herrera <[email protected]>
@ 2024-04-18 20:00 ` Alexander Lakhin <[email protected]>
2024-04-22 10:22 ` Re: cataloguing NOT NULL constraints Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 92+ messages in thread
From: Alexander Lakhin @ 2024-04-18 20:00 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Andrew Bille <[email protected]>; Peter Eisentraut <[email protected]>; Robert Haas <[email protected]>; [email protected]
Hello Alvaro,
18.04.2024 16:39, Alvaro Herrera wrote:
> I have pushed a fix which should hopefully fix this problem
> (d9f686a72e). Please give this a look. Thanks for reporting the issue.
Please look at an assertion failure, introduced with d9f686a72:
CREATE TABLE t(a int, NOT NULL a NO INHERIT);
CREATE TABLE t2() INHERITS (t);
ALTER TABLE t ADD CONSTRAINT nna NOT NULL a;
TRAP: failed Assert("lockmode != NoLock || IsBootstrapProcessingMode() || CheckRelationLockedByMe(r, AccessShareLock,
true)"), File: "relation.c", Line: 67, PID: 2980258
On d9f686a72~1 this script results in:
ERROR: cannot change NO INHERIT status of inherited NOT NULL constraint "t_a_not_null" on relation "t"
Best regards,
Alexander
^ permalink raw reply [nested|flat] 92+ messages in thread
* Re: cataloguing NOT NULL constraints
2024-04-18 13:39 Re: cataloguing NOT NULL constraints Alvaro Herrera <[email protected]>
2024-04-18 20:00 ` Re: cataloguing NOT NULL constraints Alexander Lakhin <[email protected]>
@ 2024-04-22 10:22 ` Alvaro Herrera <[email protected]>
0 siblings, 0 replies; 92+ messages in thread
From: Alvaro Herrera @ 2024-04-22 10:22 UTC (permalink / raw)
To: Alexander Lakhin <[email protected]>; +Cc: Dean Rasheed <[email protected]>; Andrew Bille <[email protected]>; Peter Eisentraut <[email protected]>; Robert Haas <[email protected]>; [email protected]
Hi Alexander,
On 2024-Apr-18, Alexander Lakhin wrote:
> 18.04.2024 16:39, Alvaro Herrera wrote:
> > I have pushed a fix which should hopefully fix this problem
> > (d9f686a72e). Please give this a look. Thanks for reporting the issue.
>
> Please look at an assertion failure, introduced with d9f686a72:
> CREATE TABLE t(a int, NOT NULL a NO INHERIT);
> CREATE TABLE t2() INHERITS (t);
>
> ALTER TABLE t ADD CONSTRAINT nna NOT NULL a;
> TRAP: failed Assert("lockmode != NoLock || IsBootstrapProcessingMode() ||
> CheckRelationLockedByMe(r, AccessShareLock, true)"), File: "relation.c",
> Line: 67, PID: 2980258
Ah, of course -- we're missing acquiring locks during the prep phase for
the recursive case of ADD CONSTRAINT. So we just need to add
find_all_inheritors() to do so in the AT_AddConstraint case in
ATPrepCmd(). However these naked find_all_inheritors() call look a bit
ugly to me, so I couldn't resist the temptation of adding a static
function ATLockAllDescendants to clean it up a bit. I'll also add your
script to the tests and push shortly.
> On d9f686a72~1 this script results in:
> ERROR: cannot change NO INHERIT status of inherited NOT NULL constraint "t_a_not_null" on relation "t"
Right. Now I'm beginning to wonder if allowing ADD CONSTRAINT to mutate
a pre-existing NO INHERIT constraint into a inheritable constraint
(while accepting a constraint name in the command that we don't heed) is
really what we want. Maybe we should throw some error when the affected
constraint is the topmost one, and only accept the inheritance status
change when we're recursing.
Also I just noticed that in 9b581c534186 (which introduced this error
message) I used ERRCODE_DATATYPE_MISMATCH ... Is that really appropriate
here?
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"Once again, thank you and all of the developers for your hard work on
PostgreSQL. This is by far the most pleasant management experience of
any database I've worked on." (Dan Harris)
http://archives.postgresql.org/pgsql-performance/2006-04/msg00247.php
Attachments:
[text/x-diff] 0001-Acquire-locks-on-children-before-recursing.patch (3.8K, ../../[email protected]/2-0001-Acquire-locks-on-children-before-recursing.patch)
download | inline diff:
From 35b485b72d0675d631cde9f95e65d9c0db9254b8 Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <[email protected]>
Date: Mon, 22 Apr 2024 11:32:04 +0200
Subject: [PATCH] Acquire locks on children before recursing
ALTER TABLE ADD CONSTRAINT was missing this, as evidenced by assertion
failures.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
src/backend/commands/tablecmds.c | 33 ++++++++++++++++++++++----------
1 file changed, 23 insertions(+), 10 deletions(-)
diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c
index 3556240c8e..8941181912 100644
--- a/src/backend/commands/tablecmds.c
+++ b/src/backend/commands/tablecmds.c
@@ -427,6 +427,7 @@ static void ATSimplePermissions(AlterTableType cmdtype, Relation rel, int allowe
static void ATSimpleRecursion(List **wqueue, Relation rel,
AlterTableCmd *cmd, bool recurse, LOCKMODE lockmode,
AlterTableUtilityContext *context);
+static void ATLockAllDescendants(Oid relid, LOCKMODE lockmode);
static void ATCheckPartitionsNotInUse(Relation rel, LOCKMODE lockmode);
static void ATTypedTableRecursion(List **wqueue, Relation rel, AlterTableCmd *cmd,
LOCKMODE lockmode,
@@ -1621,9 +1622,7 @@ RemoveRelations(DropStmt *drop)
* will lock those objects in the other order.
*/
if (state.actual_relkind == RELKIND_PARTITIONED_INDEX)
- (void) find_all_inheritors(state.heapOid,
- state.heap_lockmode,
- NULL);
+ ATLockAllDescendants(state.heapOid, state.heap_lockmode);
/* OK, we're ready to delete this one */
obj.classId = RelationRelationId;
@@ -4979,10 +4978,15 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd,
break;
case AT_AddConstraint: /* ADD CONSTRAINT */
ATSimplePermissions(cmd->subtype, rel, ATT_TABLE | ATT_FOREIGN_TABLE);
- /* Recursion occurs during execution phase */
- /* No command-specific prep needed except saving recurse flag */
if (recurse)
+ {
+ /*
+ * Make note for execution phase about need for recursion;
+ * also acquire lock on all descendants.
+ */
+ ATLockAllDescendants(RelationGetRelid(rel), lockmode);
cmd->recurse = true;
+ }
pass = AT_PASS_ADD_CONSTR;
break;
case AT_AddIndexConstraint: /* ADD CONSTRAINT USING INDEX */
@@ -6721,6 +6725,17 @@ ATSimpleRecursion(List **wqueue, Relation rel,
}
}
+/*
+ * ATLockAllDescendants
+ *
+ * Acquire lock on all descendants of the given relation.
+ */
+static void
+ATLockAllDescendants(Oid relid, LOCKMODE lockmode)
+{
+ (void) find_all_inheritors(relid, lockmode, NULL);
+}
+
/*
* Obtain list of partitions of the given table, locking them all at the given
* lockmode and ensuring that they all pass CheckTableNotInUse.
@@ -9370,10 +9385,9 @@ ATPrepAddPrimaryKey(List **wqueue, Relation rel, AlterTableCmd *cmd,
/*
* Acquire locks all the way down the hierarchy. The recursion to lower
- * levels occurs at execution time as necessary, so we don't need to do it
- * here, and we don't need the returned list either.
+ * levels occurs at execution time as necessary.
*/
- (void) find_all_inheritors(RelationGetRelid(rel), lockmode, NULL);
+ ATLockAllDescendants(RelationGetRelid(rel), lockmode);
/*
* Construct the list of constraints that we need to add to each child
@@ -11258,8 +11272,7 @@ CloneFkReferencing(List **wqueue, Relation parentRel, Relation partRel)
*/
pkrel = table_open(constrForm->confrelid, ShareRowExclusiveLock);
if (pkrel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE)
- (void) find_all_inheritors(RelationGetRelid(pkrel),
- ShareRowExclusiveLock, NULL);
+ ATLockAllDescendants(RelationGetRelid(pkrel), ShareRowExclusiveLock);
DeconstructFkConstraintRow(tuple, &numfks, conkey, confkey,
conpfeqop, conppeqop, conffeqop,
--
2.39.2
^ permalink raw reply [nested|flat] 92+ messages in thread
end of thread, other threads:[~2024-04-22 10:22 UTC | newest]
Thread overview: 92+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v10 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v8 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v5 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v4 1/2] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v7 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2021-01-13 10:21 [PATCH v6 1/3] Move a few ResourceOwnerEnlarge() calls for safety and clarity. Heikki Linnakangas <[email protected]>
2024-04-18 13:39 Re: cataloguing NOT NULL constraints Alvaro Herrera <[email protected]>
2024-04-18 20:00 ` Re: cataloguing NOT NULL constraints Alexander Lakhin <[email protected]>
2024-04-22 10:22 ` Re: cataloguing NOT NULL constraints Alvaro Herrera <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox