public inbox for [email protected]
help / color / mirror / Atom feedFrom: David Rowley <[email protected]>
To: Andres Freund <[email protected]>
Cc: [email protected]
Cc: Robert Haas <[email protected]>
Cc: Michael Paquier <[email protected]>
Cc: Tomas Vondra <[email protected]>
Subject: Re: Avoid stack frame setup in performance critical routines using tail calls
Date: Mon, 26 Feb 2024 20:42:34 +1300
Message-ID: <CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<CAApHDvrZmQ6RvJ1KDj_v30FmTmAgegtmf=Woxx7HYM0g05wQHw@mail.gmail.com>
<CAApHDvp9dA8KSEr7oE2dQn-tG6faH2Mtc3qnFgNghwNvBoq8Nw@mail.gmail.com>
<CAApHDvqHUJTtAha=wcLBAXkz6PNkMAFzsPzTHXig+Q3TUrQcug@mail.gmail.com>
<[email protected]>
On Fri, 23 Feb 2024 at 11:53, Andres Freund <[email protected]> wrote:
> > @@ -1061,6 +1072,16 @@ MemoryContextAlloc(MemoryContext context, Size size)
> >
> > context->isReset = false;
> >
>
> For a moment this made me wonder if we could move the isReset handling into
> the allocator slow paths as well - it's annoying to write that bit (and thus
> dirty the cacheline) over and ove. But it'd be somewhat awkward due to
> pre-allocated blocks. So that'd be a larger change better done separately.
It makes sense to do this, but on looking closer for aset.c, it seems
like the only time we can avoid un-setting the isReset flag is when
allocating from the freelist. We must unset it for large allocations
and for allocations that don't fit onto the existing block (the
exiting block could be the keeper block) and for allocations that
require a new block.
With the current arrangement of code in generation.c, I didn't see any
path we could skip doing context->isReset = false.
For slab.c, it's very easy and we can skip setting the isReset in most cases.
I've attached the patches I benchmarked against 449e798c7 and also the
patch I used to add a function to exercise palloc.
The query I ran was:
select chksz,mtype,pg_allocate_memory_test_reset(chksz, 1024*1024,
1024*1024*1024, mtype)
from (values(8),(16),(32),(64)) sizes(chksz),
(values('aset'),('generation'),('slab')) cxt(mtype)
order by mtype,chksz;
David
From 14c1a7dd73c8b011cb2c986aaefdc7588a44c84a Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 18 Jul 2023 18:55:58 -0700
Subject: [PATCH v4 1/3] Optimize palloc() etc to allow sibling calls
The reason palloc() etc previously couldn't use sibling calls is that they did
check the returned value (to e.g. raise an error when the allocation
fails). Push the error handling down into the memory context implementation -
they can avoid performing the check at all in the hot code paths.
---
src/backend/utils/mmgr/alignedalloc.c | 11 +-
src/backend/utils/mmgr/aset.c | 23 ++-
src/backend/utils/mmgr/generation.c | 14 +-
src/backend/utils/mmgr/mcxt.c | 233 +++++++++++---------------
src/backend/utils/mmgr/slab.c | 11 +-
src/include/nodes/memnodes.h | 42 ++++-
src/include/utils/memutils_internal.h | 30 +++-
7 files changed, 199 insertions(+), 165 deletions(-)
diff --git a/src/backend/utils/mmgr/alignedalloc.c b/src/backend/utils/mmgr/alignedalloc.c
index 7204fe64ae..c266fb3dbb 100644
--- a/src/backend/utils/mmgr/alignedalloc.c
+++ b/src/backend/utils/mmgr/alignedalloc.c
@@ -57,7 +57,7 @@ AlignedAllocFree(void *pointer)
* memory will be uninitialized.
*/
void *
-AlignedAllocRealloc(void *pointer, Size size)
+AlignedAllocRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *redirchunk = PointerGetMemoryChunk(pointer);
Size alignto;
@@ -97,14 +97,17 @@ AlignedAllocRealloc(void *pointer, Size size)
#endif
ctx = GetMemoryChunkContext(unaligned);
- newptr = MemoryContextAllocAligned(ctx, size, alignto, 0);
+ newptr = MemoryContextAllocAligned(ctx, size, alignto, flags);
/*
* We may memcpy beyond the end of the original allocation request size,
* so we must mark the entire allocation as defined.
*/
- VALGRIND_MAKE_MEM_DEFINED(pointer, old_size);
- memcpy(newptr, pointer, Min(size, old_size));
+ if (likely(newptr != NULL))
+ {
+ VALGRIND_MAKE_MEM_DEFINED(pointer, old_size);
+ memcpy(newptr, pointer, Min(size, old_size));
+ }
pfree(unaligned);
return newptr;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 2f99fa9a2f..81c3120c2b 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -700,7 +700,7 @@ AllocSetDelete(MemoryContext context)
* return space that is marked NOACCESS - AllocSetRealloc has to beware!
*/
void *
-AllocSetAlloc(MemoryContext context, Size size)
+AllocSetAlloc(MemoryContext context, Size size, int flags)
{
AllocSet set = (AllocSet) context;
AllocBlock block;
@@ -717,6 +717,9 @@ AllocSetAlloc(MemoryContext context, Size size)
*/
if (size > set->allocChunkLimit)
{
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize(context, size, flags);
+
#ifdef MEMORY_CONTEXT_CHECKING
/* ensure there's always space for the sentinel byte */
chunk_size = MAXALIGN(size + 1);
@@ -727,7 +730,7 @@ AllocSetAlloc(MemoryContext context, Size size)
blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
block = (AllocBlock) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -940,7 +943,7 @@ AllocSetAlloc(MemoryContext context, Size size)
}
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -1106,7 +1109,7 @@ AllocSetFree(void *pointer)
* request size.)
*/
void *
-AllocSetRealloc(void *pointer, Size size)
+AllocSetRealloc(void *pointer, Size size, int flags)
{
AllocBlock block;
AllocSet set;
@@ -1139,6 +1142,9 @@ AllocSetRealloc(void *pointer, Size size)
set = block->aset;
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize((MemoryContext) set, size, flags);
+
oldchksize = block->endptr - (char *) pointer;
#ifdef MEMORY_CONTEXT_CHECKING
@@ -1165,7 +1171,8 @@ AllocSetRealloc(void *pointer, Size size)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure(&set->header, size, flags);
+
}
/* updated separately, not to underflow when (oldblksize > blksize) */
@@ -1325,15 +1332,15 @@ AllocSetRealloc(void *pointer, Size size)
AllocPointer newPointer;
Size oldsize;
- /* allocate new chunk */
- newPointer = AllocSetAlloc((MemoryContext) set, size);
+ /* allocate new chunk (also checks size for us) */
+ newPointer = AllocSetAlloc((MemoryContext) set, size, flags);
/* leave immediately if request was not completed */
if (newPointer == NULL)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure((MemoryContext) set, size, flags);
}
/*
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index f9016a7ed7..4b4155b863 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -344,7 +344,7 @@ GenerationDelete(MemoryContext context)
* return space that is marked NOACCESS - GenerationRealloc has to beware!
*/
void *
-GenerationAlloc(MemoryContext context, Size size)
+GenerationAlloc(MemoryContext context, Size size, int flags)
{
GenerationContext *set = (GenerationContext *) context;
GenerationBlock *block;
@@ -367,9 +367,11 @@ GenerationAlloc(MemoryContext context, Size size)
{
Size blksize = required_size + Generation_BLOCKHDRSZ;
+ MemoryContextCheckSize((MemoryContext) set, size, flags);
+
block = (GenerationBlock *) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -472,7 +474,7 @@ GenerationAlloc(MemoryContext context, Size size)
block = (GenerationBlock *) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -737,7 +739,7 @@ GenerationFree(void *pointer)
* into the old chunk - in that case we just update chunk header.
*/
void *
-GenerationRealloc(void *pointer, Size size)
+GenerationRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
GenerationContext *set;
@@ -840,14 +842,14 @@ GenerationRealloc(void *pointer, Size size)
}
/* allocate new chunk */
- newPointer = GenerationAlloc((MemoryContext) set, size);
+ newPointer = GenerationAlloc((MemoryContext) set, size, flags);
/* leave immediately if request was not completed */
if (newPointer == NULL)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, Generation_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure((MemoryContext) set, size, flags);
}
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index ad7409a02c..c214d323c8 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -34,7 +34,7 @@
static void BogusFree(void *pointer);
-static void *BogusRealloc(void *pointer, Size size);
+static void *BogusRealloc(void *pointer, Size size, int flags);
static MemoryContext BogusGetChunkContext(void *pointer);
static Size BogusGetChunkSpace(void *pointer);
@@ -237,7 +237,7 @@ BogusFree(void *pointer)
}
static void *
-BogusRealloc(void *pointer, Size size)
+BogusRealloc(void *pointer, Size size, int flags)
{
elog(ERROR, "repalloc called with invalid pointer %p (header 0x%016llx)",
pointer, (unsigned long long) GetMemoryChunkHeader(pointer));
@@ -1023,6 +1023,38 @@ MemoryContextCreate(MemoryContext node,
VALGRIND_CREATE_MEMPOOL(node, 0, false);
}
+/*
+ * MemoryContextAllocationFailure
+ * For use by MemoryContextMethods implementations to handle when malloc
+ * returns NULL. The bahavior is specific to whether MCXT_ALLOC_NO_OOM
+ * is in 'flags'.
+ */
+void *
+MemoryContextAllocationFailure(MemoryContext context, Size size, int flags)
+{
+ if ((flags & MCXT_ALLOC_NO_OOM) == 0)
+ {
+ MemoryContextStats(TopMemoryContext);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory"),
+ errdetail("Failed on request of size %zu in memory context \"%s\".",
+ size, context->name)));
+ }
+ return NULL;
+}
+
+/*
+ * MemoryContextSizeFailure
+ * For use by MemoryContextMethods implementations to handle invalid
+ * memory allocation request sizes.
+ */
+void
+MemoryContextSizeFailure(MemoryContext context, Size size, int flags)
+{
+ elog(ERROR, "invalid memory alloc request size %zu", size);
+}
+
/*
* MemoryContextAlloc
* Allocate space within the specified context.
@@ -1038,28 +1070,19 @@ MemoryContextAlloc(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
-
- /*
- * Here, and elsewhere in this module, we show the target context's
- * "name" but not its "ident" (if any) in user-visible error messages.
- * The "ident" string might contain security-sensitive data, such as
- * values in SQL commands.
- */
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1081,24 +1104,16 @@ MemoryContextAllocZero(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
+ /*
+ * XXX: Should this also be moved into alloc()? We could possibly avoid
+ * zeroing in some cases (e.g. if we used mmap() ourselves.
+ */
MemSetAligned(ret, 0, size);
return ret;
@@ -1122,20 +1137,9 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
context->isReset = false;
- ret = context->methods->alloc(context, size);
+ ret = context->methods->alloc(context, size, flags);
if (unlikely(ret == NULL))
- {
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
return NULL;
- }
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1207,22 +1211,21 @@ palloc(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
-
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, 0);
+ /* We expect OOM to be handled by the alloc function */
+ Assert(ret != NULL);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
@@ -1238,21 +1241,9 @@ palloc0(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1271,24 +1262,11 @@ palloc_extended(Size size, int flags)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!((flags & MCXT_ALLOC_HUGE) != 0 ? AllocHugeSizeIsValid(size) :
- AllocSizeIsValid(size)))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
+ ret = context->methods->alloc(context, size, flags);
if (unlikely(ret == NULL))
{
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
return NULL;
}
@@ -1458,26 +1436,22 @@ repalloc(void *pointer, Size size)
#endif
void *ret;
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
AssertNotInCriticalSection(context);
/* isReset must be false already */
Assert(!context->isReset);
- ret = MCXT_METHOD(pointer, realloc) (pointer, size);
- if (unlikely(ret == NULL))
- {
- MemoryContext cxt = GetMemoryChunkContext(pointer);
-
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, cxt->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'realloc'
+ * function instead.
+ */
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size, 0);
#ifdef USE_VALGRIND
if (method != MCTX_ALIGNED_REDIRECT_ID)
@@ -1500,31 +1474,24 @@ repalloc_extended(void *pointer, Size size, int flags)
#endif
void *ret;
- if (!((flags & MCXT_ALLOC_HUGE) != 0 ? AllocHugeSizeIsValid(size) :
- AllocSizeIsValid(size)))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
AssertNotInCriticalSection(context);
/* isReset must be false already */
Assert(!context->isReset);
- ret = MCXT_METHOD(pointer, realloc) (pointer, size);
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'realloc'
+ * function instead.
+ */
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size, flags);
if (unlikely(ret == NULL))
- {
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContext cxt = GetMemoryChunkContext(pointer);
-
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, cxt->name)));
- }
return NULL;
- }
VALGRIND_MEMPOOL_CHANGE(context, pointer, ret, size);
@@ -1565,21 +1532,19 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocHugeSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, MCXT_ALLOC_HUGE);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index b8f00cfc7c..bc91446cb3 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -496,7 +496,7 @@ SlabDelete(MemoryContext context)
* request could not be completed; memory is added to the slab.
*/
void *
-SlabAlloc(MemoryContext context, Size size)
+SlabAlloc(MemoryContext context, Size size, int flags)
{
SlabContext *slab = (SlabContext *) context;
SlabBlock *block;
@@ -508,7 +508,10 @@ SlabAlloc(MemoryContext context, Size size)
Assert(slab->curBlocklistIndex >= 0);
Assert(slab->curBlocklistIndex <= SlabBlocklistIndex(slab, slab->chunksPerBlock));
- /* make sure we only allow correct request size */
+ /*
+ * Make sure we only allow correct request size. This doubles as the
+ * MemoryContextCheckSize check.
+ */
if (unlikely(size != slab->chunkSize))
elog(ERROR, "unexpected alloc chunk size %zu (expected %u)",
size, slab->chunkSize);
@@ -546,7 +549,7 @@ SlabAlloc(MemoryContext context, Size size)
block = (SlabBlock *) malloc(slab->blockSize);
if (unlikely(block == NULL))
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
block->slab = slab;
context->mem_allocated += slab->blockSize;
@@ -770,7 +773,7 @@ SlabFree(void *pointer)
* realloc is usually used to enlarge the chunk.
*/
void *
-SlabRealloc(void *pointer, Size size)
+SlabRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
SlabBlock *block;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index a48f7e5a18..edc0257f36 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -57,20 +57,58 @@ typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
typedef struct MemoryContextMethods
{
- void *(*alloc) (MemoryContext context, Size size);
+ /*
+ * Function to handle memory allocation requests of 'size' to allocate
+ * memory into the given 'context'. The function must handle flags
+ * MCXT_ALLOC_HUGE and MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by
+ * the calling function.
+ */
+ void *(*alloc) (MemoryContext context, Size size, int flags);
+
/* call this free_p in case someone #define's free() */
void (*free_p) (void *pointer);
- void *(*realloc) (void *pointer, Size size);
+
+ /*
+ * Function to handle a size change request for an existing allocation.
+ * The implementation must handle flags MCXT_ALLOC_HUGE and
+ * MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by the calling function.
+ */
+ void *(*realloc) (void *pointer, Size size, int flags);
+
+ /*
+ * Invalidate all previous allocations in the given memory context and
+ * prepare the context for a new set of allocations. Implementations may
+ * optionally free() excess memory back to the OS during this time.
+ */
void (*reset) (MemoryContext context);
+
+ /* Free all memory consumed by the given MemoryContext. */
void (*delete_context) (MemoryContext context);
+
+ /* Return the MemoryContext that the given pointer belongs to. */
MemoryContext (*get_chunk_context) (void *pointer);
+
+ /*
+ * Return the number of bytes consumed by the given pointer within its
+ * memory context, including the overhead of alignment and chunk headers.
+ */
Size (*get_chunk_space) (void *pointer);
+
+ /*
+ * Return true if the given MemoryContext has not had any allocations
+ * since it was created or last reset.
+ */
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
MemoryContextCounters *totals,
bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
+
+ /*
+ * Perform validation checks on the given context and raise any discovered
+ * anomalies as WARNINGs.
+ */
void (*check) (MemoryContext context);
#endif
} MemoryContextMethods;
diff --git a/src/include/utils/memutils_internal.h b/src/include/utils/memutils_internal.h
index e0c4f3d5af..ad1048fd82 100644
--- a/src/include/utils/memutils_internal.h
+++ b/src/include/utils/memutils_internal.h
@@ -19,9 +19,9 @@
#include "utils/memutils.h"
/* These functions implement the MemoryContext API for AllocSet context. */
-extern void *AllocSetAlloc(MemoryContext context, Size size);
+extern void *AllocSetAlloc(MemoryContext context, Size size, int flags);
extern void AllocSetFree(void *pointer);
-extern void *AllocSetRealloc(void *pointer, Size size);
+extern void *AllocSetRealloc(void *pointer, Size size, int flags);
extern void AllocSetReset(MemoryContext context);
extern void AllocSetDelete(MemoryContext context);
extern MemoryContext AllocSetGetChunkContext(void *pointer);
@@ -36,9 +36,9 @@ extern void AllocSetCheck(MemoryContext context);
#endif
/* These functions implement the MemoryContext API for Generation context. */
-extern void *GenerationAlloc(MemoryContext context, Size size);
+extern void *GenerationAlloc(MemoryContext context, Size size, int flags);
extern void GenerationFree(void *pointer);
-extern void *GenerationRealloc(void *pointer, Size size);
+extern void *GenerationRealloc(void *pointer, Size size, int flags);
extern void GenerationReset(MemoryContext context);
extern void GenerationDelete(MemoryContext context);
extern MemoryContext GenerationGetChunkContext(void *pointer);
@@ -54,9 +54,9 @@ extern void GenerationCheck(MemoryContext context);
/* These functions implement the MemoryContext API for Slab context. */
-extern void *SlabAlloc(MemoryContext context, Size size);
+extern void *SlabAlloc(MemoryContext context, Size size, int flags);
extern void SlabFree(void *pointer);
-extern void *SlabRealloc(void *pointer, Size size);
+extern void *SlabRealloc(void *pointer, Size size, int flags);
extern void SlabReset(MemoryContext context);
extern void SlabDelete(MemoryContext context);
extern MemoryContext SlabGetChunkContext(void *pointer);
@@ -75,7 +75,7 @@ extern void SlabCheck(MemoryContext context);
* part of a fully-fledged MemoryContext type.
*/
extern void AlignedAllocFree(void *pointer);
-extern void *AlignedAllocRealloc(void *pointer, Size size);
+extern void *AlignedAllocRealloc(void *pointer, Size size, int flags);
extern MemoryContext AlignedAllocGetChunkContext(void *pointer);
extern Size AlignedAllocGetChunkSpace(void *pointer);
@@ -133,4 +133,20 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void *MemoryContextAllocationFailure(MemoryContext context, Size size,
+ int flags);
+
+extern void MemoryContextSizeFailure(MemoryContext context, Size size,
+ int flags) pg_attribute_noreturn();
+
+static inline void
+MemoryContextCheckSize(MemoryContext context, Size size, int flags)
+{
+ if (unlikely(!AllocSizeIsValid(size)))
+ {
+ if (!(flags & MCXT_ALLOC_HUGE) || !AllocHugeSizeIsValid(size))
+ MemoryContextSizeFailure(context, size, flags);
+ }
+}
+
#endif /* MEMUTILS_INTERNAL_H */
--
2.40.1.windows.1
From 239295dc3dca9ad1f05a71a2169c89405f4ce041 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 18 Jul 2023 20:16:18 -0700
Subject: [PATCH v4 2/3] Optimize AllocSetAlloc() by separating hot from cold
paths.
With gcc the common paths of AllocSetAlloc() now don't need to initialize a
stack frame when compiling with gcc.
---
src/backend/utils/mmgr/aset.c | 482 +++++++++++++++++++---------------
1 file changed, 269 insertions(+), 213 deletions(-)
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 81c3120c2b..4e52eb063c 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -687,101 +687,301 @@ AllocSetDelete(MemoryContext context)
}
/*
- * AllocSetAlloc
- * Returns pointer to allocated memory of given size or NULL if
- * request could not be completed; memory is added to the set.
+ * Helper for AllocSetAlloc() that allocates an entire block for the chunk.
*
- * No request may exceed:
- * MAXALIGN_DOWN(SIZE_MAX) - ALLOC_BLOCKHDRSZ - ALLOC_CHUNKHDRSZ
- * All callers use a much-lower limit.
- *
- * Note: when using valgrind, it doesn't matter how the returned allocation
- * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
- * return space that is marked NOACCESS - AllocSetRealloc has to beware!
+ * AllocSetAlloc()'s comment explains why this is separate.
*/
-void *
-AllocSetAlloc(MemoryContext context, Size size, int flags)
+pg_noinline
+static void *
+AllocSetAllocLarge(MemoryContext context, Size size, int flags)
{
AllocSet set = (AllocSet) context;
- AllocBlock block;
+ AllocBlock block;
MemoryChunk *chunk;
- int fidx;
- Size chunk_size;
- Size blksize;
+ Size chunk_size;
+ Size blksize;
- Assert(AllocSetIsValid(set));
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize(context, size, flags);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+ /* ensure there's always space for the sentinel byte */
+ chunk_size = MAXALIGN(size + 1);
+#else
+ chunk_size = MAXALIGN(size);
+#endif
+
+ blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
+ block = (AllocBlock) malloc(blksize);
+ if (block == NULL)
+ return MemoryContextAllocationFailure(context, size, flags);
+
+ context->mem_allocated += blksize;
+
+ block->aset = set;
+ block->freeptr = block->endptr = ((char *) block) + blksize;
+
+ chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+
+ /* mark the MemoryChunk as externally managed */
+ MemoryChunkSetHdrMaskExternal(chunk, MCTX_ASET_ID);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+ chunk->requested_size = size;
+ /* set mark to catch clobber of "unused" space */
+ Assert(size < chunk_size);
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
+#endif
+#ifdef RANDOMIZE_ALLOCATED_MEMORY
+ /* fill the allocated space with junk */
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
+#endif
/*
- * If requested size exceeds maximum for chunks, allocate an entire block
- * for this request.
+ * Stick the new block underneath the active allocation block, if any,
+ * so that we don't lose the use of the space remaining therein.
*/
- if (size > set->allocChunkLimit)
+ if (set->blocks != NULL)
{
- /* check size, only allocation path where the limits could be hit */
- MemoryContextCheckSize(context, size, flags);
+ block->prev = set->blocks;
+ block->next = set->blocks->next;
+ if (block->next)
+ block->next->prev = block;
+ set->blocks->next = block;
+ }
+ else
+ {
+ block->prev = NULL;
+ block->next = NULL;
+ set->blocks = block;
+ }
-#ifdef MEMORY_CONTEXT_CHECKING
- /* ensure there's always space for the sentinel byte */
- chunk_size = MAXALIGN(size + 1);
-#else
- chunk_size = MAXALIGN(size);
-#endif
+ /* Ensure any padding bytes are marked NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
+ chunk_size - size);
- blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
- block = (AllocBlock) malloc(blksize);
- if (block == NULL)
- return MemoryContextAllocationFailure(context, size, flags);
+ /* Disallow access to the chunk header. */
+ VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- context->mem_allocated += blksize;
+ return MemoryChunkGetPointer(chunk);
+}
- block->aset = set;
- block->freeptr = block->endptr = ((char *) block) + blksize;
+/*
+ * Small helper for allocating a new chunk from a chunk, to avoid duplicating
+ * the code between AllocSetAlloc() and AllocSetAllocFromNewBlock().
+ */
+static inline void *
+AllocSetAllocChunkFromBlock(MemoryContext context, AllocBlock block,
+ Size size, Size chunk_size, int fidx)
+{
+ MemoryChunk *chunk;
- chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+ chunk = (MemoryChunk *) (block->freeptr);
- /* mark the MemoryChunk as externally managed */
- MemoryChunkSetHdrMaskExternal(chunk, MCTX_ASET_ID);
+ /* Prepare to initialize the chunk header. */
+ VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
+
+ block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
+ Assert(block->freeptr <= block->endptr);
+
+ /* store the free list index in the value field */
+ MemoryChunkSetHdrMask(chunk, block, fidx, MCTX_ASET_ID);
#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = size;
- /* set mark to catch clobber of "unused" space */
- Assert(size < chunk_size);
+ chunk->requested_size = size;
+ /* set mark to catch clobber of "unused" space */
+ if (size < chunk_size)
set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
- /* fill the allocated space with junk */
- randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
+ /* fill the allocated space with junk */
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
+ /* Ensure any padding bytes are marked NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
+ chunk_size - size);
+
+ /* Disallow access to the chunk header. */
+ VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
+
+ return MemoryChunkGetPointer(chunk);
+}
+
+/*
+ * Helper for AllocSetAlloc() that allocates a new block and returns a chunk
+ * allocated from it.
+ *
+ * AllocSetAlloc()'s comment explains why this is separate.
+ */
+pg_noinline
+static void *
+AllocSetAllocFromNewBlock(MemoryContext context, Size size, int flags,
+ int fidx)
+{
+ AllocSet set = (AllocSet) context;
+ AllocBlock block;
+ Size availspace;
+ Size blksize;
+ Size required_size;
+ Size chunk_size;
+
+ /* due to the keeper block set->blocks should always be valid */
+ Assert(set->blocks != NULL);
+ block = set->blocks;
+ availspace = block->endptr - block->freeptr;
+
+ /*
+ * The existing active (top) block does not have enough room for
+ * the requested allocation, but it might still have a useful
+ * amount of space in it. Once we push it down in the block list,
+ * we'll never try to allocate more space from it. So, before we
+ * do that, carve up its free space into chunks that we can put on
+ * the set's freelists.
+ *
+ * Because we can only get here when there's less than
+ * ALLOC_CHUNK_LIMIT left in the block, this loop cannot iterate
+ * more than ALLOCSET_NUM_FREELISTS-1 times.
+ */
+ while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ))
+ {
+ AllocFreeListLink *link;
+ Size availchunk = availspace - ALLOC_CHUNKHDRSZ;
+ int a_fidx = AllocSetFreeIndex(availchunk);
+ MemoryChunk *chunk;
+
/*
- * Stick the new block underneath the active allocation block, if any,
- * so that we don't lose the use of the space remaining therein.
+ * In most cases, we'll get back the index of the next larger
+ * freelist than the one we need to put this chunk on. The
+ * exception is when availchunk is exactly a power of 2.
*/
- if (set->blocks != NULL)
+ if (availchunk != GetChunkSizeFromFreeListIdx(a_fidx))
{
- block->prev = set->blocks;
- block->next = set->blocks->next;
- if (block->next)
- block->next->prev = block;
- set->blocks->next = block;
- }
- else
- {
- block->prev = NULL;
- block->next = NULL;
- set->blocks = block;
+ a_fidx--;
+ Assert(a_fidx >= 0);
+ availchunk = GetChunkSizeFromFreeListIdx(a_fidx);
}
- /* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
- chunk_size - size);
+ chunk = (MemoryChunk *) (block->freeptr);
- /* Disallow access to the chunk header. */
- VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
+ /* Prepare to initialize the chunk header. */
+ VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
+ block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ);
+ availspace -= (availchunk + ALLOC_CHUNKHDRSZ);
- return MemoryChunkGetPointer(chunk);
+ /* store the freelist index in the value field */
+ MemoryChunkSetHdrMask(chunk, block, a_fidx, MCTX_ASET_ID);
+#ifdef MEMORY_CONTEXT_CHECKING
+ chunk->requested_size = InvalidAllocSize; /* mark it free */
+#endif
+ /* push this chunk onto the free list */
+ link = GetFreeListLink(chunk);
+
+ VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
+ link->next = set->freelist[a_fidx];
+ VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));
+
+ set->freelist[a_fidx] = chunk;
}
+ /*
+ * The first such block has size initBlockSize, and we double the
+ * space in each succeeding block, but not more than maxBlockSize.
+ */
+ blksize = set->nextBlockSize;
+ set->nextBlockSize <<= 1;
+ if (set->nextBlockSize > set->maxBlockSize)
+ set->nextBlockSize = set->maxBlockSize;
+
+ chunk_size = GetChunkSizeFromFreeListIdx(fidx);
+
+ /*
+ * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more
+ * space... but try to keep it a power of 2.
+ */
+ required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
+ while (blksize < required_size)
+ blksize <<= 1;
+
+ /* Try to allocate it */
+ block = (AllocBlock) malloc(blksize);
+
+ /*
+ * We could be asking for pretty big blocks here, so cope if malloc
+ * fails. But give up if there's less than 1 MB or so available...
+ */
+ while (block == NULL && blksize > 1024 * 1024)
+ {
+ blksize >>= 1;
+ if (blksize < required_size)
+ break;
+ block = (AllocBlock) malloc(blksize);
+ }
+
+ if (block == NULL)
+ return MemoryContextAllocationFailure(context, size, flags);
+
+ context->mem_allocated += blksize;
+
+ block->aset = set;
+ block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ;
+ block->endptr = ((char *) block) + blksize;
+
+ /* Mark unallocated space NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS(block->freeptr,
+ blksize - ALLOC_BLOCKHDRSZ);
+
+ block->prev = NULL;
+ block->next = set->blocks;
+ if (block->next)
+ block->next->prev = block;
+ set->blocks = block;
+
+ return AllocSetAllocChunkFromBlock(context, block, size, chunk_size, fidx);
+}
+
+/*
+ * AllocSetAlloc
+ * Returns pointer to allocated memory of given size or NULL if
+ * request could not be completed; memory is added to the set.
+ *
+ * No request may exceed:
+ * MAXALIGN_DOWN(SIZE_MAX) - ALLOC_BLOCKHDRSZ - ALLOC_CHUNKHDRSZ
+ * All callers use a much-lower limit.
+ *
+ * Note: when using valgrind, it doesn't matter how the returned allocation
+ * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
+ * return space that is marked NOACCESS - AllocSetRealloc has to beware!
+ *
+ * This function should only contain the most common code paths, everything
+ * else should be in pg_noinline helper functions. That allows to avoid the
+ * overhead of creating a stack frame for the common cases - this function is
+ * one of the most common bottlenecks, making this worthwhile. The helper
+ * functions should always directly return the newly allocated memory,
+ * otherwise the stack frame is required after all.
+ */
+void *
+AllocSetAlloc(MemoryContext context, Size size, int flags)
+{
+ AllocSet set = (AllocSet) context;
+ AllocBlock block;
+ MemoryChunk *chunk;
+ int fidx;
+ Size chunk_size;
+ Size availspace;
+
+ Assert(AllocSetIsValid(set));
+
+ /* due to the keeper block set->blocks should always be valid */
+ Assert(set->blocks != NULL);
+
+ /*
+ * If requested size exceeds maximum for chunks, allocate an entire block
+ * for this request.
+ */
+ if (size > set->allocChunkLimit)
+ return AllocSetAllocLarge(context, size, flags);
+
/*
* Request is small enough to be treated as a chunk. Look in the
* corresponding free list to see if there is a free chunk we could reuse.
@@ -837,164 +1037,20 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
chunk_size = GetChunkSizeFromFreeListIdx(fidx);
Assert(chunk_size >= size);
+ block = set->blocks;
+ availspace = block->endptr - block->freeptr;
+
/*
* If there is enough room in the active allocation block, we will put the
* chunk into that block. Else must start a new one.
*/
- if ((block = set->blocks) != NULL)
- {
- Size availspace = block->endptr - block->freeptr;
-
- if (availspace < (chunk_size + ALLOC_CHUNKHDRSZ))
- {
- /*
- * The existing active (top) block does not have enough room for
- * the requested allocation, but it might still have a useful
- * amount of space in it. Once we push it down in the block list,
- * we'll never try to allocate more space from it. So, before we
- * do that, carve up its free space into chunks that we can put on
- * the set's freelists.
- *
- * Because we can only get here when there's less than
- * ALLOC_CHUNK_LIMIT left in the block, this loop cannot iterate
- * more than ALLOCSET_NUM_FREELISTS-1 times.
- */
- while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ))
- {
- AllocFreeListLink *link;
- Size availchunk = availspace - ALLOC_CHUNKHDRSZ;
- int a_fidx = AllocSetFreeIndex(availchunk);
-
- /*
- * In most cases, we'll get back the index of the next larger
- * freelist than the one we need to put this chunk on. The
- * exception is when availchunk is exactly a power of 2.
- */
- if (availchunk != GetChunkSizeFromFreeListIdx(a_fidx))
- {
- a_fidx--;
- Assert(a_fidx >= 0);
- availchunk = GetChunkSizeFromFreeListIdx(a_fidx);
- }
-
- chunk = (MemoryChunk *) (block->freeptr);
-
- /* Prepare to initialize the chunk header. */
- VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
- block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ);
- availspace -= (availchunk + ALLOC_CHUNKHDRSZ);
-
- /* store the freelist index in the value field */
- MemoryChunkSetHdrMask(chunk, block, a_fidx, MCTX_ASET_ID);
-#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = InvalidAllocSize; /* mark it free */
-#endif
- /* push this chunk onto the free list */
- link = GetFreeListLink(chunk);
-
- VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
- link->next = set->freelist[a_fidx];
- VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));
-
- set->freelist[a_fidx] = chunk;
- }
- /* Mark that we need to create a new block */
- block = NULL;
- }
- }
-
- /*
- * Time to create a new regular (multi-chunk) block?
- */
- if (block == NULL)
- {
- Size required_size;
-
- /*
- * The first such block has size initBlockSize, and we double the
- * space in each succeeding block, but not more than maxBlockSize.
- */
- blksize = set->nextBlockSize;
- set->nextBlockSize <<= 1;
- if (set->nextBlockSize > set->maxBlockSize)
- set->nextBlockSize = set->maxBlockSize;
-
- /*
- * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more
- * space... but try to keep it a power of 2.
- */
- required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
- while (blksize < required_size)
- blksize <<= 1;
-
- /* Try to allocate it */
- block = (AllocBlock) malloc(blksize);
-
- /*
- * We could be asking for pretty big blocks here, so cope if malloc
- * fails. But give up if there's less than 1 MB or so available...
- */
- while (block == NULL && blksize > 1024 * 1024)
- {
- blksize >>= 1;
- if (blksize < required_size)
- break;
- block = (AllocBlock) malloc(blksize);
- }
-
- if (block == NULL)
- return MemoryContextAllocationFailure(context, size, flags);
-
- context->mem_allocated += blksize;
-
- block->aset = set;
- block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ;
- block->endptr = ((char *) block) + blksize;
-
- /* Mark unallocated space NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS(block->freeptr,
- blksize - ALLOC_BLOCKHDRSZ);
-
- block->prev = NULL;
- block->next = set->blocks;
- if (block->next)
- block->next->prev = block;
- set->blocks = block;
- }
+ if (unlikely(availspace < (chunk_size + ALLOC_CHUNKHDRSZ)))
+ return AllocSetAllocFromNewBlock(context, size, flags, fidx);
/*
* OK, do the allocation
*/
- chunk = (MemoryChunk *) (block->freeptr);
-
- /* Prepare to initialize the chunk header. */
- VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
-
- block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
- Assert(block->freeptr <= block->endptr);
-
- /* store the free list index in the value field */
- MemoryChunkSetHdrMask(chunk, block, fidx, MCTX_ASET_ID);
-
-#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = size;
- /* set mark to catch clobber of "unused" space */
- if (size < chunk_size)
- set_sentinel(MemoryChunkGetPointer(chunk), size);
-#endif
-#ifdef RANDOMIZE_ALLOCATED_MEMORY
- /* fill the allocated space with junk */
- randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
-#endif
-
- /* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
- chunk_size - size);
-
- /* Disallow access to the chunk header. */
- VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
-
- return MemoryChunkGetPointer(chunk);
+ return AllocSetAllocChunkFromBlock(context, block, size, chunk_size, fidx);
}
/*
--
2.40.1.windows.1
From c69840cd85767fa7ee4d1688a605a0978c3fc39f Mon Sep 17 00:00:00 2001
From: David Rowley <[email protected]>
Date: Mon, 26 Feb 2024 20:05:34 +1300
Subject: [PATCH v4 3/3] Make unsetting the isReset flag the MemoryContext's
job
Up until now, setting this boolean flag to false has been the job of the
allocation function in mcxt.c. Making this the job of the MemoryContext
implementation can make things more efficient as adjusting the flag can
just be put in code paths that are only possible to reach in cases where
we may not have already done an allocation since the context was created
or reset.
Discussion: https://postgr.es/m/[email protected]
---
src/backend/utils/mmgr/aset.c | 11 ++++++++++
src/backend/utils/mmgr/generation.c | 2 ++
src/backend/utils/mmgr/mcxt.c | 31 ++++++++++++++++-------------
src/backend/utils/mmgr/slab.c | 14 +++++++++++++
src/include/nodes/memnodes.h | 5 ++++-
5 files changed, 48 insertions(+), 15 deletions(-)
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 4e52eb063c..a1c59381d5 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -711,6 +711,7 @@ AllocSetAllocLarge(MemoryContext context, Size size, int flags)
chunk_size = MAXALIGN(size);
#endif
+ context->isReset = false;
blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
block = (AllocBlock) malloc(blksize);
if (block == NULL)
@@ -1000,6 +1001,9 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
{
AllocFreeListLink *link = GetFreeListLink(chunk);
+ /* should already be unset if we've something in the freelist */
+ Assert(context->isReset == false);
+
/* Allow access to the chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOC_CHUNKHDRSZ);
@@ -1040,6 +1044,13 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
block = set->blocks;
availspace = block->endptr - block->freeptr;
+ /*
+ * We must un-reset the context here as lack of space for this allocation
+ * on a block is no guarantee that we've already seen an allocation as
+ * the given block could be the keeper block.
+ */
+ context->isReset = false;
+
/*
* If there is enough room in the active allocation block, we will put the
* chunk into that block. Else must start a new one.
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 4b4155b863..5a351c26df 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -352,6 +352,8 @@ GenerationAlloc(MemoryContext context, Size size, int flags)
Size chunk_size;
Size required_size;
+ context->isReset = false;
+
Assert(GenerationIsValid(set));
#ifdef MEMORY_CONTEXT_CHECKING
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index c214d323c8..01f3bec028 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1070,8 +1070,6 @@ MemoryContextAlloc(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1084,6 +1082,8 @@ MemoryContextAlloc(MemoryContext context, Size size)
*/
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
@@ -1104,10 +1104,10 @@ MemoryContextAllocZero(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
/*
@@ -1135,9 +1135,10 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
AllocSizeIsValid(size)))
elog(ERROR, "invalid memory alloc request size %zu", size);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, flags);
+
+ Assert(!context->isReset);
+
if (unlikely(ret == NULL))
return NULL;
@@ -1211,8 +1212,6 @@ palloc(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1224,6 +1223,9 @@ palloc(Size size)
* function instead.
*/
ret = context->methods->alloc(context, size, 0);
+
+ Assert(!context->isReset);
+
/* We expect OOM to be handled by the alloc function */
Assert(ret != NULL);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1241,10 +1243,10 @@ palloc0(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
MemSetAligned(ret, 0, size);
@@ -1262,9 +1264,10 @@ palloc_extended(Size size, int flags)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, flags);
+
+ Assert(!context->isReset);
+
if (unlikely(ret == NULL))
{
return NULL;
@@ -1532,8 +1535,6 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1546,6 +1547,8 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
*/
ret = context->methods->alloc(context, size, MCXT_ALLOC_HUGE);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index bc91446cb3..1160aa7e20 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -533,6 +533,12 @@ SlabAlloc(MemoryContext context, Size size, int flags)
{
dlist_node *node = dclist_pop_head_node(&slab->emptyblocks);
+ /*
+ * We should already have already un-reset the context if we've
+ * got a block on the emptyblocks list.
+ */
+ Assert(!context->isReset);
+
block = dlist_container(SlabBlock, node, node);
/*
@@ -546,6 +552,8 @@ SlabAlloc(MemoryContext context, Size size, int flags)
}
else
{
+ context->isReset = false;
+
block = (SlabBlock *) malloc(slab->blockSize);
if (unlikely(block == NULL))
@@ -581,6 +589,12 @@ SlabAlloc(MemoryContext context, Size size, int flags)
Assert(!dlist_is_empty(blocklist));
+ /*
+ * We should already have already un-reset the context if we've got a
+ * non-zero curBlocklistIndex
+ */
+ Assert(!context->isReset);
+
/* grab the block from the blocklist */
block = dlist_head_element(SlabBlock, node, blocklist);
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index edc0257f36..72539cfc5d 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -61,7 +61,10 @@ typedef struct MemoryContextMethods
* Function to handle memory allocation requests of 'size' to allocate
* memory into the given 'context'. The function must handle flags
* MCXT_ALLOC_HUGE and MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by
- * the calling function.
+ * the calling function. The implementation must handle setting
+ * MemoryContext.isReset to false before returning. Leaving this up to
+ * the implementing function can allow this flag to be unset more
+ * efficiently.
*/
void *(*alloc) (MemoryContext context, Size size, int flags);
--
2.40.1.windows.1
From 519ba53c785e1858952c2fa2692dc34aed54b76a Mon Sep 17 00:00:00 2001
From: David Rowley <[email protected]>
Date: Wed, 25 May 2022 14:24:37 +1200
Subject: [PATCH v99] Function to test palloc/pfree performance
---
src/backend/utils/adt/mcxtfuncs.c | 213 ++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 12 ++
2 files changed, 225 insertions(+)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 4708d73f5f..416235b473 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,6 +15,8 @@
#include "postgres.h"
+#include <time.h>
+
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
@@ -193,3 +195,214 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+typedef struct AllocateTestNext
+{
+ struct AllocateTestNext *next; /* ptr to the next allocation */
+} AllocateTestNext;
+
+/* #define ALLOCATE_TEST_DEBUG */
+/*
+ * pg_allocate_memory_test
+ * Used to test the performance of a memory context types
+ */
+Datum
+pg_allocate_memory_test(PG_FUNCTION_ARGS)
+{
+ int32 chunk_size = PG_GETARG_INT32(0);
+ int64 keep_memory = PG_GETARG_INT64(1);
+ int64 total_alloc = PG_GETARG_INT64(2);
+ text *context_type_text = PG_GETARG_TEXT_PP(3);
+ char *context_type;
+ int64 curr_memory_use = 0;
+ int64 remaining_alloc_bytes = total_alloc;
+ MemoryContext context;
+ MemoryContext oldContext;
+ AllocateTestNext *next_free_ptr = NULL;
+ AllocateTestNext *last_alloc = NULL;
+ clock_t start, end;
+
+ if (chunk_size < sizeof(AllocateTestNext))
+ elog(ERROR, "chunk_size (%d) must be at least %ld bytes", chunk_size,
+ sizeof(AllocateTestNext));
+ if (keep_memory > total_alloc)
+ elog(ERROR, "keep_memory (" INT64_FORMAT ") must be less than total_alloc (" INT64_FORMAT ")",
+ keep_memory, total_alloc);
+
+ context_type = text_to_cstring(context_type_text);
+
+ start = clock();
+
+ if (strcmp(context_type, "generation") == 0)
+ context = GenerationContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "aset") == 0)
+ context = AllocSetContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "slab") == 0)
+ context = SlabContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_MAXSIZE,
+ chunk_size);
+ else
+ elog(ERROR, "context_type must be \"generation\", \"aset\" or \"slab\"");
+
+ oldContext = MemoryContextSwitchTo(context);
+
+ while (remaining_alloc_bytes > 0)
+ {
+ AllocateTestNext *curr_alloc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Allocate the memory and update the counters */
+ curr_alloc = (AllocateTestNext *) palloc(chunk_size);
+ remaining_alloc_bytes -= chunk_size;
+ curr_memory_use += chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "alloc %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", curr_alloc, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ /*
+ * Point the last allocate to this one so that we can free allocations
+ * starting with the oldest first.
+ */
+ curr_alloc->next = NULL;
+ if (last_alloc != NULL)
+ last_alloc->next = curr_alloc;
+
+ if (next_free_ptr == NULL)
+ {
+ /*
+ * Remember the first chunk to free. We will follow the ->next
+ * pointers to find the next chunk to free when freeing memory
+ */
+ next_free_ptr = curr_alloc;
+ }
+
+ /*
+ * If the currently allocated memory has reached or exceeded the amount
+ * of memory we want to keep allocated at once then we'd better free
+ * some. Since all allocations are the same size we only need to free
+ * one allocation per loop.
+ */
+ if (curr_memory_use >= keep_memory)
+ {
+ AllocateTestNext *next = next_free_ptr->next;
+
+ /* free the memory and update the current memory usage */
+ pfree(next_free_ptr);
+ curr_memory_use -= chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "free %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", next_free_ptr, curr_memory_use, remaining_alloc_bytes);
+#endif
+ /* get the next chunk to free */
+ next_free_ptr = next;
+ }
+
+ if (curr_memory_use > 0)
+ last_alloc = curr_alloc;
+ else
+ last_alloc = NULL;
+ }
+
+ /* cleanup loop -- pfree remaining memory */
+ while (next_free_ptr != NULL)
+ {
+ AllocateTestNext *next = next_free_ptr->next;
+
+ /* free the memory and update the current memory usage */
+ pfree(next_free_ptr);
+ curr_memory_use -= chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "free %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", next_free_ptr, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ next_free_ptr = next;
+ }
+
+ MemoryContextSwitchTo(oldContext);
+
+ end = clock();
+
+ PG_RETURN_FLOAT8((double) (end - start) / CLOCKS_PER_SEC);
+}
+
+Datum
+pg_allocate_memory_test_reset(PG_FUNCTION_ARGS)
+{
+ int32 chunk_size = PG_GETARG_INT32(0);
+ int64 keep_memory = PG_GETARG_INT64(1);
+ int64 total_alloc = PG_GETARG_INT64(2);
+ text *context_type_text = PG_GETARG_TEXT_PP(3);
+ char *context_type;
+ int64 curr_memory_use = 0;
+ int64 remaining_alloc_bytes = total_alloc;
+ MemoryContext context;
+ MemoryContext oldContext;
+ clock_t start, end;
+
+ if (chunk_size < 1)
+ elog(ERROR, "size of chunk must be above 0");
+ if (keep_memory > total_alloc)
+ elog(ERROR, "keep_memory (" INT64_FORMAT ") must be less than total_alloc (" INT64_FORMAT ")",
+ keep_memory, total_alloc);
+
+ context_type = text_to_cstring(context_type_text);
+
+ start = clock();
+
+ if (strcmp(context_type, "generation") == 0)
+ context = GenerationContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "aset") == 0)
+ context = AllocSetContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "slab") == 0)
+ context = SlabContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_MAXSIZE,
+ chunk_size);
+ else
+ elog(ERROR, "context_type must be \"generation\", \"aset\" or \"slab\"");
+
+ oldContext = MemoryContextSwitchTo(context);
+
+ while (remaining_alloc_bytes > 0)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ /* Allocate the memory and update the counters */
+ (void) palloc(chunk_size);
+ remaining_alloc_bytes -= chunk_size;
+ curr_memory_use += chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "alloc %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", curr_alloc, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ /*
+ * If the currently allocated memory has reached or exceeded the amount
+ * of memory we want to keep allocated at once then reset the context.
+ */
+ if (curr_memory_use >= keep_memory)
+ {
+ curr_memory_use = 0;
+ MemoryContextReset(context);
+ }
+ }
+
+ MemoryContextSwitchTo(oldContext);
+ MemoryContextDelete(context);
+
+ end = clock();
+
+ PG_RETURN_FLOAT8((double) (end - start) / CLOCKS_PER_SEC);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9c120fc2b7..f307229c60 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8274,6 +8274,18 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# just for testing memory context allocation speed
+{ oid => '9319', descr => 'for testing performance of allocation and freeing',
+ proname => 'pg_allocate_memory_test', provolatile => 'v',
+ prorettype => 'float8', proargtypes => 'int4 int8 int8 text',
+ prosrc => 'pg_allocate_memory_test' },
+
+# just for testing memory context allocation speed
+{ oid => '9320', descr => 'for testing performance of allocation and resetting',
+ proname => 'pg_allocate_memory_test_reset', provolatile => 'v',
+ prorettype => 'float8', proargtypes => 'int4 int8 int8 text',
+ prosrc => 'pg_allocate_memory_test_reset' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
--
2.40.1.windows.1
Attachments:
[text/plain] v4-0001-Optimize-palloc-etc-to-allow-sibling-calls.patch (25.4K, ../CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com/2-v4-0001-Optimize-palloc-etc-to-allow-sibling-calls.patch)
download | inline diff:
From 14c1a7dd73c8b011cb2c986aaefdc7588a44c84a Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 18 Jul 2023 18:55:58 -0700
Subject: [PATCH v4 1/3] Optimize palloc() etc to allow sibling calls
The reason palloc() etc previously couldn't use sibling calls is that they did
check the returned value (to e.g. raise an error when the allocation
fails). Push the error handling down into the memory context implementation -
they can avoid performing the check at all in the hot code paths.
---
src/backend/utils/mmgr/alignedalloc.c | 11 +-
src/backend/utils/mmgr/aset.c | 23 ++-
src/backend/utils/mmgr/generation.c | 14 +-
src/backend/utils/mmgr/mcxt.c | 233 +++++++++++---------------
src/backend/utils/mmgr/slab.c | 11 +-
src/include/nodes/memnodes.h | 42 ++++-
src/include/utils/memutils_internal.h | 30 +++-
7 files changed, 199 insertions(+), 165 deletions(-)
diff --git a/src/backend/utils/mmgr/alignedalloc.c b/src/backend/utils/mmgr/alignedalloc.c
index 7204fe64ae..c266fb3dbb 100644
--- a/src/backend/utils/mmgr/alignedalloc.c
+++ b/src/backend/utils/mmgr/alignedalloc.c
@@ -57,7 +57,7 @@ AlignedAllocFree(void *pointer)
* memory will be uninitialized.
*/
void *
-AlignedAllocRealloc(void *pointer, Size size)
+AlignedAllocRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *redirchunk = PointerGetMemoryChunk(pointer);
Size alignto;
@@ -97,14 +97,17 @@ AlignedAllocRealloc(void *pointer, Size size)
#endif
ctx = GetMemoryChunkContext(unaligned);
- newptr = MemoryContextAllocAligned(ctx, size, alignto, 0);
+ newptr = MemoryContextAllocAligned(ctx, size, alignto, flags);
/*
* We may memcpy beyond the end of the original allocation request size,
* so we must mark the entire allocation as defined.
*/
- VALGRIND_MAKE_MEM_DEFINED(pointer, old_size);
- memcpy(newptr, pointer, Min(size, old_size));
+ if (likely(newptr != NULL))
+ {
+ VALGRIND_MAKE_MEM_DEFINED(pointer, old_size);
+ memcpy(newptr, pointer, Min(size, old_size));
+ }
pfree(unaligned);
return newptr;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 2f99fa9a2f..81c3120c2b 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -700,7 +700,7 @@ AllocSetDelete(MemoryContext context)
* return space that is marked NOACCESS - AllocSetRealloc has to beware!
*/
void *
-AllocSetAlloc(MemoryContext context, Size size)
+AllocSetAlloc(MemoryContext context, Size size, int flags)
{
AllocSet set = (AllocSet) context;
AllocBlock block;
@@ -717,6 +717,9 @@ AllocSetAlloc(MemoryContext context, Size size)
*/
if (size > set->allocChunkLimit)
{
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize(context, size, flags);
+
#ifdef MEMORY_CONTEXT_CHECKING
/* ensure there's always space for the sentinel byte */
chunk_size = MAXALIGN(size + 1);
@@ -727,7 +730,7 @@ AllocSetAlloc(MemoryContext context, Size size)
blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
block = (AllocBlock) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -940,7 +943,7 @@ AllocSetAlloc(MemoryContext context, Size size)
}
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -1106,7 +1109,7 @@ AllocSetFree(void *pointer)
* request size.)
*/
void *
-AllocSetRealloc(void *pointer, Size size)
+AllocSetRealloc(void *pointer, Size size, int flags)
{
AllocBlock block;
AllocSet set;
@@ -1139,6 +1142,9 @@ AllocSetRealloc(void *pointer, Size size)
set = block->aset;
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize((MemoryContext) set, size, flags);
+
oldchksize = block->endptr - (char *) pointer;
#ifdef MEMORY_CONTEXT_CHECKING
@@ -1165,7 +1171,8 @@ AllocSetRealloc(void *pointer, Size size)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure(&set->header, size, flags);
+
}
/* updated separately, not to underflow when (oldblksize > blksize) */
@@ -1325,15 +1332,15 @@ AllocSetRealloc(void *pointer, Size size)
AllocPointer newPointer;
Size oldsize;
- /* allocate new chunk */
- newPointer = AllocSetAlloc((MemoryContext) set, size);
+ /* allocate new chunk (also checks size for us) */
+ newPointer = AllocSetAlloc((MemoryContext) set, size, flags);
/* leave immediately if request was not completed */
if (newPointer == NULL)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure((MemoryContext) set, size, flags);
}
/*
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index f9016a7ed7..4b4155b863 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -344,7 +344,7 @@ GenerationDelete(MemoryContext context)
* return space that is marked NOACCESS - GenerationRealloc has to beware!
*/
void *
-GenerationAlloc(MemoryContext context, Size size)
+GenerationAlloc(MemoryContext context, Size size, int flags)
{
GenerationContext *set = (GenerationContext *) context;
GenerationBlock *block;
@@ -367,9 +367,11 @@ GenerationAlloc(MemoryContext context, Size size)
{
Size blksize = required_size + Generation_BLOCKHDRSZ;
+ MemoryContextCheckSize((MemoryContext) set, size, flags);
+
block = (GenerationBlock *) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -472,7 +474,7 @@ GenerationAlloc(MemoryContext context, Size size)
block = (GenerationBlock *) malloc(blksize);
if (block == NULL)
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
context->mem_allocated += blksize;
@@ -737,7 +739,7 @@ GenerationFree(void *pointer)
* into the old chunk - in that case we just update chunk header.
*/
void *
-GenerationRealloc(void *pointer, Size size)
+GenerationRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
GenerationContext *set;
@@ -840,14 +842,14 @@ GenerationRealloc(void *pointer, Size size)
}
/* allocate new chunk */
- newPointer = GenerationAlloc((MemoryContext) set, size);
+ newPointer = GenerationAlloc((MemoryContext) set, size, flags);
/* leave immediately if request was not completed */
if (newPointer == NULL)
{
/* Disallow access to the chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, Generation_CHUNKHDRSZ);
- return NULL;
+ return MemoryContextAllocationFailure((MemoryContext) set, size, flags);
}
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index ad7409a02c..c214d323c8 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -34,7 +34,7 @@
static void BogusFree(void *pointer);
-static void *BogusRealloc(void *pointer, Size size);
+static void *BogusRealloc(void *pointer, Size size, int flags);
static MemoryContext BogusGetChunkContext(void *pointer);
static Size BogusGetChunkSpace(void *pointer);
@@ -237,7 +237,7 @@ BogusFree(void *pointer)
}
static void *
-BogusRealloc(void *pointer, Size size)
+BogusRealloc(void *pointer, Size size, int flags)
{
elog(ERROR, "repalloc called with invalid pointer %p (header 0x%016llx)",
pointer, (unsigned long long) GetMemoryChunkHeader(pointer));
@@ -1023,6 +1023,38 @@ MemoryContextCreate(MemoryContext node,
VALGRIND_CREATE_MEMPOOL(node, 0, false);
}
+/*
+ * MemoryContextAllocationFailure
+ * For use by MemoryContextMethods implementations to handle when malloc
+ * returns NULL. The bahavior is specific to whether MCXT_ALLOC_NO_OOM
+ * is in 'flags'.
+ */
+void *
+MemoryContextAllocationFailure(MemoryContext context, Size size, int flags)
+{
+ if ((flags & MCXT_ALLOC_NO_OOM) == 0)
+ {
+ MemoryContextStats(TopMemoryContext);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory"),
+ errdetail("Failed on request of size %zu in memory context \"%s\".",
+ size, context->name)));
+ }
+ return NULL;
+}
+
+/*
+ * MemoryContextSizeFailure
+ * For use by MemoryContextMethods implementations to handle invalid
+ * memory allocation request sizes.
+ */
+void
+MemoryContextSizeFailure(MemoryContext context, Size size, int flags)
+{
+ elog(ERROR, "invalid memory alloc request size %zu", size);
+}
+
/*
* MemoryContextAlloc
* Allocate space within the specified context.
@@ -1038,28 +1070,19 @@ MemoryContextAlloc(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
-
- /*
- * Here, and elsewhere in this module, we show the target context's
- * "name" but not its "ident" (if any) in user-visible error messages.
- * The "ident" string might contain security-sensitive data, such as
- * values in SQL commands.
- */
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1081,24 +1104,16 @@ MemoryContextAllocZero(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
+ /*
+ * XXX: Should this also be moved into alloc()? We could possibly avoid
+ * zeroing in some cases (e.g. if we used mmap() ourselves.
+ */
MemSetAligned(ret, 0, size);
return ret;
@@ -1122,20 +1137,9 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
context->isReset = false;
- ret = context->methods->alloc(context, size);
+ ret = context->methods->alloc(context, size, flags);
if (unlikely(ret == NULL))
- {
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
return NULL;
- }
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1207,22 +1211,21 @@ palloc(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
-
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, 0);
+ /* We expect OOM to be handled by the alloc function */
+ Assert(ret != NULL);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
@@ -1238,21 +1241,9 @@ palloc0(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ ret = context->methods->alloc(context, size, 0);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1271,24 +1262,11 @@ palloc_extended(Size size, int flags)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!((flags & MCXT_ALLOC_HUGE) != 0 ? AllocHugeSizeIsValid(size) :
- AllocSizeIsValid(size)))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
+ ret = context->methods->alloc(context, size, flags);
if (unlikely(ret == NULL))
{
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
return NULL;
}
@@ -1458,26 +1436,22 @@ repalloc(void *pointer, Size size)
#endif
void *ret;
- if (!AllocSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
AssertNotInCriticalSection(context);
/* isReset must be false already */
Assert(!context->isReset);
- ret = MCXT_METHOD(pointer, realloc) (pointer, size);
- if (unlikely(ret == NULL))
- {
- MemoryContext cxt = GetMemoryChunkContext(pointer);
-
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, cxt->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'realloc'
+ * function instead.
+ */
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size, 0);
#ifdef USE_VALGRIND
if (method != MCTX_ALIGNED_REDIRECT_ID)
@@ -1500,31 +1474,24 @@ repalloc_extended(void *pointer, Size size, int flags)
#endif
void *ret;
- if (!((flags & MCXT_ALLOC_HUGE) != 0 ? AllocHugeSizeIsValid(size) :
- AllocSizeIsValid(size)))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
AssertNotInCriticalSection(context);
/* isReset must be false already */
Assert(!context->isReset);
- ret = MCXT_METHOD(pointer, realloc) (pointer, size);
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'realloc'
+ * function instead.
+ */
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size, flags);
if (unlikely(ret == NULL))
- {
- if ((flags & MCXT_ALLOC_NO_OOM) == 0)
- {
- MemoryContext cxt = GetMemoryChunkContext(pointer);
-
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, cxt->name)));
- }
return NULL;
- }
VALGRIND_MEMPOOL_CHANGE(context, pointer, ret, size);
@@ -1565,21 +1532,19 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- if (!AllocHugeSizeIsValid(size))
- elog(ERROR, "invalid memory alloc request size %zu", size);
-
context->isReset = false;
- ret = context->methods->alloc(context, size);
- if (unlikely(ret == NULL))
- {
- MemoryContextStats(TopMemoryContext);
- ereport(ERROR,
- (errcode(ERRCODE_OUT_OF_MEMORY),
- errmsg("out of memory"),
- errdetail("Failed on request of size %zu in memory context \"%s\".",
- size, context->name)));
- }
+ /*
+ * For efficiency reasons, we purposefully offload the handling of
+ * allocation failures to the MemoryContextMethods implementation as this
+ * allows these checks to be performed only when an actual malloc needs to
+ * be done to request more memory from the OS. Additionally, not having
+ * to execute any instructions after this call allows the compiler to use
+ * the tailcall optimization. If you're considering adding code after
+ * this call, consider making it the responsibility of the 'alloc'
+ * function instead.
+ */
+ ret = context->methods->alloc(context, size, MCXT_ALLOC_HUGE);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index b8f00cfc7c..bc91446cb3 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -496,7 +496,7 @@ SlabDelete(MemoryContext context)
* request could not be completed; memory is added to the slab.
*/
void *
-SlabAlloc(MemoryContext context, Size size)
+SlabAlloc(MemoryContext context, Size size, int flags)
{
SlabContext *slab = (SlabContext *) context;
SlabBlock *block;
@@ -508,7 +508,10 @@ SlabAlloc(MemoryContext context, Size size)
Assert(slab->curBlocklistIndex >= 0);
Assert(slab->curBlocklistIndex <= SlabBlocklistIndex(slab, slab->chunksPerBlock));
- /* make sure we only allow correct request size */
+ /*
+ * Make sure we only allow correct request size. This doubles as the
+ * MemoryContextCheckSize check.
+ */
if (unlikely(size != slab->chunkSize))
elog(ERROR, "unexpected alloc chunk size %zu (expected %u)",
size, slab->chunkSize);
@@ -546,7 +549,7 @@ SlabAlloc(MemoryContext context, Size size)
block = (SlabBlock *) malloc(slab->blockSize);
if (unlikely(block == NULL))
- return NULL;
+ return MemoryContextAllocationFailure(context, size, flags);
block->slab = slab;
context->mem_allocated += slab->blockSize;
@@ -770,7 +773,7 @@ SlabFree(void *pointer)
* realloc is usually used to enlarge the chunk.
*/
void *
-SlabRealloc(void *pointer, Size size)
+SlabRealloc(void *pointer, Size size, int flags)
{
MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
SlabBlock *block;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index a48f7e5a18..edc0257f36 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -57,20 +57,58 @@ typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
typedef struct MemoryContextMethods
{
- void *(*alloc) (MemoryContext context, Size size);
+ /*
+ * Function to handle memory allocation requests of 'size' to allocate
+ * memory into the given 'context'. The function must handle flags
+ * MCXT_ALLOC_HUGE and MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by
+ * the calling function.
+ */
+ void *(*alloc) (MemoryContext context, Size size, int flags);
+
/* call this free_p in case someone #define's free() */
void (*free_p) (void *pointer);
- void *(*realloc) (void *pointer, Size size);
+
+ /*
+ * Function to handle a size change request for an existing allocation.
+ * The implementation must handle flags MCXT_ALLOC_HUGE and
+ * MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by the calling function.
+ */
+ void *(*realloc) (void *pointer, Size size, int flags);
+
+ /*
+ * Invalidate all previous allocations in the given memory context and
+ * prepare the context for a new set of allocations. Implementations may
+ * optionally free() excess memory back to the OS during this time.
+ */
void (*reset) (MemoryContext context);
+
+ /* Free all memory consumed by the given MemoryContext. */
void (*delete_context) (MemoryContext context);
+
+ /* Return the MemoryContext that the given pointer belongs to. */
MemoryContext (*get_chunk_context) (void *pointer);
+
+ /*
+ * Return the number of bytes consumed by the given pointer within its
+ * memory context, including the overhead of alignment and chunk headers.
+ */
Size (*get_chunk_space) (void *pointer);
+
+ /*
+ * Return true if the given MemoryContext has not had any allocations
+ * since it was created or last reset.
+ */
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
MemoryContextCounters *totals,
bool print_to_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
+
+ /*
+ * Perform validation checks on the given context and raise any discovered
+ * anomalies as WARNINGs.
+ */
void (*check) (MemoryContext context);
#endif
} MemoryContextMethods;
diff --git a/src/include/utils/memutils_internal.h b/src/include/utils/memutils_internal.h
index e0c4f3d5af..ad1048fd82 100644
--- a/src/include/utils/memutils_internal.h
+++ b/src/include/utils/memutils_internal.h
@@ -19,9 +19,9 @@
#include "utils/memutils.h"
/* These functions implement the MemoryContext API for AllocSet context. */
-extern void *AllocSetAlloc(MemoryContext context, Size size);
+extern void *AllocSetAlloc(MemoryContext context, Size size, int flags);
extern void AllocSetFree(void *pointer);
-extern void *AllocSetRealloc(void *pointer, Size size);
+extern void *AllocSetRealloc(void *pointer, Size size, int flags);
extern void AllocSetReset(MemoryContext context);
extern void AllocSetDelete(MemoryContext context);
extern MemoryContext AllocSetGetChunkContext(void *pointer);
@@ -36,9 +36,9 @@ extern void AllocSetCheck(MemoryContext context);
#endif
/* These functions implement the MemoryContext API for Generation context. */
-extern void *GenerationAlloc(MemoryContext context, Size size);
+extern void *GenerationAlloc(MemoryContext context, Size size, int flags);
extern void GenerationFree(void *pointer);
-extern void *GenerationRealloc(void *pointer, Size size);
+extern void *GenerationRealloc(void *pointer, Size size, int flags);
extern void GenerationReset(MemoryContext context);
extern void GenerationDelete(MemoryContext context);
extern MemoryContext GenerationGetChunkContext(void *pointer);
@@ -54,9 +54,9 @@ extern void GenerationCheck(MemoryContext context);
/* These functions implement the MemoryContext API for Slab context. */
-extern void *SlabAlloc(MemoryContext context, Size size);
+extern void *SlabAlloc(MemoryContext context, Size size, int flags);
extern void SlabFree(void *pointer);
-extern void *SlabRealloc(void *pointer, Size size);
+extern void *SlabRealloc(void *pointer, Size size, int flags);
extern void SlabReset(MemoryContext context);
extern void SlabDelete(MemoryContext context);
extern MemoryContext SlabGetChunkContext(void *pointer);
@@ -75,7 +75,7 @@ extern void SlabCheck(MemoryContext context);
* part of a fully-fledged MemoryContext type.
*/
extern void AlignedAllocFree(void *pointer);
-extern void *AlignedAllocRealloc(void *pointer, Size size);
+extern void *AlignedAllocRealloc(void *pointer, Size size, int flags);
extern MemoryContext AlignedAllocGetChunkContext(void *pointer);
extern Size AlignedAllocGetChunkSpace(void *pointer);
@@ -133,4 +133,20 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void *MemoryContextAllocationFailure(MemoryContext context, Size size,
+ int flags);
+
+extern void MemoryContextSizeFailure(MemoryContext context, Size size,
+ int flags) pg_attribute_noreturn();
+
+static inline void
+MemoryContextCheckSize(MemoryContext context, Size size, int flags)
+{
+ if (unlikely(!AllocSizeIsValid(size)))
+ {
+ if (!(flags & MCXT_ALLOC_HUGE) || !AllocHugeSizeIsValid(size))
+ MemoryContextSizeFailure(context, size, flags);
+ }
+}
+
#endif /* MEMUTILS_INTERNAL_H */
--
2.40.1.windows.1
[text/plain] v4-0002-Optimize-AllocSetAlloc-by-separating-hot-from-col.patch (17.6K, ../CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com/3-v4-0002-Optimize-AllocSetAlloc-by-separating-hot-from-col.patch)
download | inline diff:
From 239295dc3dca9ad1f05a71a2169c89405f4ce041 Mon Sep 17 00:00:00 2001
From: Andres Freund <[email protected]>
Date: Tue, 18 Jul 2023 20:16:18 -0700
Subject: [PATCH v4 2/3] Optimize AllocSetAlloc() by separating hot from cold
paths.
With gcc the common paths of AllocSetAlloc() now don't need to initialize a
stack frame when compiling with gcc.
---
src/backend/utils/mmgr/aset.c | 482 +++++++++++++++++++---------------
1 file changed, 269 insertions(+), 213 deletions(-)
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 81c3120c2b..4e52eb063c 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -687,101 +687,301 @@ AllocSetDelete(MemoryContext context)
}
/*
- * AllocSetAlloc
- * Returns pointer to allocated memory of given size or NULL if
- * request could not be completed; memory is added to the set.
+ * Helper for AllocSetAlloc() that allocates an entire block for the chunk.
*
- * No request may exceed:
- * MAXALIGN_DOWN(SIZE_MAX) - ALLOC_BLOCKHDRSZ - ALLOC_CHUNKHDRSZ
- * All callers use a much-lower limit.
- *
- * Note: when using valgrind, it doesn't matter how the returned allocation
- * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
- * return space that is marked NOACCESS - AllocSetRealloc has to beware!
+ * AllocSetAlloc()'s comment explains why this is separate.
*/
-void *
-AllocSetAlloc(MemoryContext context, Size size, int flags)
+pg_noinline
+static void *
+AllocSetAllocLarge(MemoryContext context, Size size, int flags)
{
AllocSet set = (AllocSet) context;
- AllocBlock block;
+ AllocBlock block;
MemoryChunk *chunk;
- int fidx;
- Size chunk_size;
- Size blksize;
+ Size chunk_size;
+ Size blksize;
- Assert(AllocSetIsValid(set));
+ /* check size, only allocation path where the limits could be hit */
+ MemoryContextCheckSize(context, size, flags);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+ /* ensure there's always space for the sentinel byte */
+ chunk_size = MAXALIGN(size + 1);
+#else
+ chunk_size = MAXALIGN(size);
+#endif
+
+ blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
+ block = (AllocBlock) malloc(blksize);
+ if (block == NULL)
+ return MemoryContextAllocationFailure(context, size, flags);
+
+ context->mem_allocated += blksize;
+
+ block->aset = set;
+ block->freeptr = block->endptr = ((char *) block) + blksize;
+
+ chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+
+ /* mark the MemoryChunk as externally managed */
+ MemoryChunkSetHdrMaskExternal(chunk, MCTX_ASET_ID);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+ chunk->requested_size = size;
+ /* set mark to catch clobber of "unused" space */
+ Assert(size < chunk_size);
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
+#endif
+#ifdef RANDOMIZE_ALLOCATED_MEMORY
+ /* fill the allocated space with junk */
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
+#endif
/*
- * If requested size exceeds maximum for chunks, allocate an entire block
- * for this request.
+ * Stick the new block underneath the active allocation block, if any,
+ * so that we don't lose the use of the space remaining therein.
*/
- if (size > set->allocChunkLimit)
+ if (set->blocks != NULL)
{
- /* check size, only allocation path where the limits could be hit */
- MemoryContextCheckSize(context, size, flags);
+ block->prev = set->blocks;
+ block->next = set->blocks->next;
+ if (block->next)
+ block->next->prev = block;
+ set->blocks->next = block;
+ }
+ else
+ {
+ block->prev = NULL;
+ block->next = NULL;
+ set->blocks = block;
+ }
-#ifdef MEMORY_CONTEXT_CHECKING
- /* ensure there's always space for the sentinel byte */
- chunk_size = MAXALIGN(size + 1);
-#else
- chunk_size = MAXALIGN(size);
-#endif
+ /* Ensure any padding bytes are marked NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
+ chunk_size - size);
- blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
- block = (AllocBlock) malloc(blksize);
- if (block == NULL)
- return MemoryContextAllocationFailure(context, size, flags);
+ /* Disallow access to the chunk header. */
+ VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
- context->mem_allocated += blksize;
+ return MemoryChunkGetPointer(chunk);
+}
- block->aset = set;
- block->freeptr = block->endptr = ((char *) block) + blksize;
+/*
+ * Small helper for allocating a new chunk from a chunk, to avoid duplicating
+ * the code between AllocSetAlloc() and AllocSetAllocFromNewBlock().
+ */
+static inline void *
+AllocSetAllocChunkFromBlock(MemoryContext context, AllocBlock block,
+ Size size, Size chunk_size, int fidx)
+{
+ MemoryChunk *chunk;
- chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+ chunk = (MemoryChunk *) (block->freeptr);
- /* mark the MemoryChunk as externally managed */
- MemoryChunkSetHdrMaskExternal(chunk, MCTX_ASET_ID);
+ /* Prepare to initialize the chunk header. */
+ VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
+
+ block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
+ Assert(block->freeptr <= block->endptr);
+
+ /* store the free list index in the value field */
+ MemoryChunkSetHdrMask(chunk, block, fidx, MCTX_ASET_ID);
#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = size;
- /* set mark to catch clobber of "unused" space */
- Assert(size < chunk_size);
+ chunk->requested_size = size;
+ /* set mark to catch clobber of "unused" space */
+ if (size < chunk_size)
set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
- /* fill the allocated space with junk */
- randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
+ /* fill the allocated space with junk */
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
+ /* Ensure any padding bytes are marked NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
+ chunk_size - size);
+
+ /* Disallow access to the chunk header. */
+ VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
+
+ return MemoryChunkGetPointer(chunk);
+}
+
+/*
+ * Helper for AllocSetAlloc() that allocates a new block and returns a chunk
+ * allocated from it.
+ *
+ * AllocSetAlloc()'s comment explains why this is separate.
+ */
+pg_noinline
+static void *
+AllocSetAllocFromNewBlock(MemoryContext context, Size size, int flags,
+ int fidx)
+{
+ AllocSet set = (AllocSet) context;
+ AllocBlock block;
+ Size availspace;
+ Size blksize;
+ Size required_size;
+ Size chunk_size;
+
+ /* due to the keeper block set->blocks should always be valid */
+ Assert(set->blocks != NULL);
+ block = set->blocks;
+ availspace = block->endptr - block->freeptr;
+
+ /*
+ * The existing active (top) block does not have enough room for
+ * the requested allocation, but it might still have a useful
+ * amount of space in it. Once we push it down in the block list,
+ * we'll never try to allocate more space from it. So, before we
+ * do that, carve up its free space into chunks that we can put on
+ * the set's freelists.
+ *
+ * Because we can only get here when there's less than
+ * ALLOC_CHUNK_LIMIT left in the block, this loop cannot iterate
+ * more than ALLOCSET_NUM_FREELISTS-1 times.
+ */
+ while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ))
+ {
+ AllocFreeListLink *link;
+ Size availchunk = availspace - ALLOC_CHUNKHDRSZ;
+ int a_fidx = AllocSetFreeIndex(availchunk);
+ MemoryChunk *chunk;
+
/*
- * Stick the new block underneath the active allocation block, if any,
- * so that we don't lose the use of the space remaining therein.
+ * In most cases, we'll get back the index of the next larger
+ * freelist than the one we need to put this chunk on. The
+ * exception is when availchunk is exactly a power of 2.
*/
- if (set->blocks != NULL)
+ if (availchunk != GetChunkSizeFromFreeListIdx(a_fidx))
{
- block->prev = set->blocks;
- block->next = set->blocks->next;
- if (block->next)
- block->next->prev = block;
- set->blocks->next = block;
- }
- else
- {
- block->prev = NULL;
- block->next = NULL;
- set->blocks = block;
+ a_fidx--;
+ Assert(a_fidx >= 0);
+ availchunk = GetChunkSizeFromFreeListIdx(a_fidx);
}
- /* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
- chunk_size - size);
+ chunk = (MemoryChunk *) (block->freeptr);
- /* Disallow access to the chunk header. */
- VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
+ /* Prepare to initialize the chunk header. */
+ VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
+ block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ);
+ availspace -= (availchunk + ALLOC_CHUNKHDRSZ);
- return MemoryChunkGetPointer(chunk);
+ /* store the freelist index in the value field */
+ MemoryChunkSetHdrMask(chunk, block, a_fidx, MCTX_ASET_ID);
+#ifdef MEMORY_CONTEXT_CHECKING
+ chunk->requested_size = InvalidAllocSize; /* mark it free */
+#endif
+ /* push this chunk onto the free list */
+ link = GetFreeListLink(chunk);
+
+ VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
+ link->next = set->freelist[a_fidx];
+ VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));
+
+ set->freelist[a_fidx] = chunk;
}
+ /*
+ * The first such block has size initBlockSize, and we double the
+ * space in each succeeding block, but not more than maxBlockSize.
+ */
+ blksize = set->nextBlockSize;
+ set->nextBlockSize <<= 1;
+ if (set->nextBlockSize > set->maxBlockSize)
+ set->nextBlockSize = set->maxBlockSize;
+
+ chunk_size = GetChunkSizeFromFreeListIdx(fidx);
+
+ /*
+ * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more
+ * space... but try to keep it a power of 2.
+ */
+ required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
+ while (blksize < required_size)
+ blksize <<= 1;
+
+ /* Try to allocate it */
+ block = (AllocBlock) malloc(blksize);
+
+ /*
+ * We could be asking for pretty big blocks here, so cope if malloc
+ * fails. But give up if there's less than 1 MB or so available...
+ */
+ while (block == NULL && blksize > 1024 * 1024)
+ {
+ blksize >>= 1;
+ if (blksize < required_size)
+ break;
+ block = (AllocBlock) malloc(blksize);
+ }
+
+ if (block == NULL)
+ return MemoryContextAllocationFailure(context, size, flags);
+
+ context->mem_allocated += blksize;
+
+ block->aset = set;
+ block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ;
+ block->endptr = ((char *) block) + blksize;
+
+ /* Mark unallocated space NOACCESS. */
+ VALGRIND_MAKE_MEM_NOACCESS(block->freeptr,
+ blksize - ALLOC_BLOCKHDRSZ);
+
+ block->prev = NULL;
+ block->next = set->blocks;
+ if (block->next)
+ block->next->prev = block;
+ set->blocks = block;
+
+ return AllocSetAllocChunkFromBlock(context, block, size, chunk_size, fidx);
+}
+
+/*
+ * AllocSetAlloc
+ * Returns pointer to allocated memory of given size or NULL if
+ * request could not be completed; memory is added to the set.
+ *
+ * No request may exceed:
+ * MAXALIGN_DOWN(SIZE_MAX) - ALLOC_BLOCKHDRSZ - ALLOC_CHUNKHDRSZ
+ * All callers use a much-lower limit.
+ *
+ * Note: when using valgrind, it doesn't matter how the returned allocation
+ * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
+ * return space that is marked NOACCESS - AllocSetRealloc has to beware!
+ *
+ * This function should only contain the most common code paths, everything
+ * else should be in pg_noinline helper functions. That allows to avoid the
+ * overhead of creating a stack frame for the common cases - this function is
+ * one of the most common bottlenecks, making this worthwhile. The helper
+ * functions should always directly return the newly allocated memory,
+ * otherwise the stack frame is required after all.
+ */
+void *
+AllocSetAlloc(MemoryContext context, Size size, int flags)
+{
+ AllocSet set = (AllocSet) context;
+ AllocBlock block;
+ MemoryChunk *chunk;
+ int fidx;
+ Size chunk_size;
+ Size availspace;
+
+ Assert(AllocSetIsValid(set));
+
+ /* due to the keeper block set->blocks should always be valid */
+ Assert(set->blocks != NULL);
+
+ /*
+ * If requested size exceeds maximum for chunks, allocate an entire block
+ * for this request.
+ */
+ if (size > set->allocChunkLimit)
+ return AllocSetAllocLarge(context, size, flags);
+
/*
* Request is small enough to be treated as a chunk. Look in the
* corresponding free list to see if there is a free chunk we could reuse.
@@ -837,164 +1037,20 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
chunk_size = GetChunkSizeFromFreeListIdx(fidx);
Assert(chunk_size >= size);
+ block = set->blocks;
+ availspace = block->endptr - block->freeptr;
+
/*
* If there is enough room in the active allocation block, we will put the
* chunk into that block. Else must start a new one.
*/
- if ((block = set->blocks) != NULL)
- {
- Size availspace = block->endptr - block->freeptr;
-
- if (availspace < (chunk_size + ALLOC_CHUNKHDRSZ))
- {
- /*
- * The existing active (top) block does not have enough room for
- * the requested allocation, but it might still have a useful
- * amount of space in it. Once we push it down in the block list,
- * we'll never try to allocate more space from it. So, before we
- * do that, carve up its free space into chunks that we can put on
- * the set's freelists.
- *
- * Because we can only get here when there's less than
- * ALLOC_CHUNK_LIMIT left in the block, this loop cannot iterate
- * more than ALLOCSET_NUM_FREELISTS-1 times.
- */
- while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ))
- {
- AllocFreeListLink *link;
- Size availchunk = availspace - ALLOC_CHUNKHDRSZ;
- int a_fidx = AllocSetFreeIndex(availchunk);
-
- /*
- * In most cases, we'll get back the index of the next larger
- * freelist than the one we need to put this chunk on. The
- * exception is when availchunk is exactly a power of 2.
- */
- if (availchunk != GetChunkSizeFromFreeListIdx(a_fidx))
- {
- a_fidx--;
- Assert(a_fidx >= 0);
- availchunk = GetChunkSizeFromFreeListIdx(a_fidx);
- }
-
- chunk = (MemoryChunk *) (block->freeptr);
-
- /* Prepare to initialize the chunk header. */
- VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
- block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ);
- availspace -= (availchunk + ALLOC_CHUNKHDRSZ);
-
- /* store the freelist index in the value field */
- MemoryChunkSetHdrMask(chunk, block, a_fidx, MCTX_ASET_ID);
-#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = InvalidAllocSize; /* mark it free */
-#endif
- /* push this chunk onto the free list */
- link = GetFreeListLink(chunk);
-
- VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreeListLink));
- link->next = set->freelist[a_fidx];
- VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreeListLink));
-
- set->freelist[a_fidx] = chunk;
- }
- /* Mark that we need to create a new block */
- block = NULL;
- }
- }
-
- /*
- * Time to create a new regular (multi-chunk) block?
- */
- if (block == NULL)
- {
- Size required_size;
-
- /*
- * The first such block has size initBlockSize, and we double the
- * space in each succeeding block, but not more than maxBlockSize.
- */
- blksize = set->nextBlockSize;
- set->nextBlockSize <<= 1;
- if (set->nextBlockSize > set->maxBlockSize)
- set->nextBlockSize = set->maxBlockSize;
-
- /*
- * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more
- * space... but try to keep it a power of 2.
- */
- required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
- while (blksize < required_size)
- blksize <<= 1;
-
- /* Try to allocate it */
- block = (AllocBlock) malloc(blksize);
-
- /*
- * We could be asking for pretty big blocks here, so cope if malloc
- * fails. But give up if there's less than 1 MB or so available...
- */
- while (block == NULL && blksize > 1024 * 1024)
- {
- blksize >>= 1;
- if (blksize < required_size)
- break;
- block = (AllocBlock) malloc(blksize);
- }
-
- if (block == NULL)
- return MemoryContextAllocationFailure(context, size, flags);
-
- context->mem_allocated += blksize;
-
- block->aset = set;
- block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ;
- block->endptr = ((char *) block) + blksize;
-
- /* Mark unallocated space NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS(block->freeptr,
- blksize - ALLOC_BLOCKHDRSZ);
-
- block->prev = NULL;
- block->next = set->blocks;
- if (block->next)
- block->next->prev = block;
- set->blocks = block;
- }
+ if (unlikely(availspace < (chunk_size + ALLOC_CHUNKHDRSZ)))
+ return AllocSetAllocFromNewBlock(context, size, flags, fidx);
/*
* OK, do the allocation
*/
- chunk = (MemoryChunk *) (block->freeptr);
-
- /* Prepare to initialize the chunk header. */
- VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
-
- block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
- Assert(block->freeptr <= block->endptr);
-
- /* store the free list index in the value field */
- MemoryChunkSetHdrMask(chunk, block, fidx, MCTX_ASET_ID);
-
-#ifdef MEMORY_CONTEXT_CHECKING
- chunk->requested_size = size;
- /* set mark to catch clobber of "unused" space */
- if (size < chunk_size)
- set_sentinel(MemoryChunkGetPointer(chunk), size);
-#endif
-#ifdef RANDOMIZE_ALLOCATED_MEMORY
- /* fill the allocated space with junk */
- randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
-#endif
-
- /* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
- chunk_size - size);
-
- /* Disallow access to the chunk header. */
- VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOC_CHUNKHDRSZ);
-
- return MemoryChunkGetPointer(chunk);
+ return AllocSetAllocChunkFromBlock(context, block, size, chunk_size, fidx);
}
/*
--
2.40.1.windows.1
[text/plain] v4-0003-Make-unsetting-the-isReset-flag-the-MemoryContext.patch (7.8K, ../CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com/4-v4-0003-Make-unsetting-the-isReset-flag-the-MemoryContext.patch)
download | inline diff:
From c69840cd85767fa7ee4d1688a605a0978c3fc39f Mon Sep 17 00:00:00 2001
From: David Rowley <[email protected]>
Date: Mon, 26 Feb 2024 20:05:34 +1300
Subject: [PATCH v4 3/3] Make unsetting the isReset flag the MemoryContext's
job
Up until now, setting this boolean flag to false has been the job of the
allocation function in mcxt.c. Making this the job of the MemoryContext
implementation can make things more efficient as adjusting the flag can
just be put in code paths that are only possible to reach in cases where
we may not have already done an allocation since the context was created
or reset.
Discussion: https://postgr.es/m/[email protected]
---
src/backend/utils/mmgr/aset.c | 11 ++++++++++
src/backend/utils/mmgr/generation.c | 2 ++
src/backend/utils/mmgr/mcxt.c | 31 ++++++++++++++++-------------
src/backend/utils/mmgr/slab.c | 14 +++++++++++++
src/include/nodes/memnodes.h | 5 ++++-
5 files changed, 48 insertions(+), 15 deletions(-)
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 4e52eb063c..a1c59381d5 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -711,6 +711,7 @@ AllocSetAllocLarge(MemoryContext context, Size size, int flags)
chunk_size = MAXALIGN(size);
#endif
+ context->isReset = false;
blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
block = (AllocBlock) malloc(blksize);
if (block == NULL)
@@ -1000,6 +1001,9 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
{
AllocFreeListLink *link = GetFreeListLink(chunk);
+ /* should already be unset if we've something in the freelist */
+ Assert(context->isReset == false);
+
/* Allow access to the chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOC_CHUNKHDRSZ);
@@ -1040,6 +1044,13 @@ AllocSetAlloc(MemoryContext context, Size size, int flags)
block = set->blocks;
availspace = block->endptr - block->freeptr;
+ /*
+ * We must un-reset the context here as lack of space for this allocation
+ * on a block is no guarantee that we've already seen an allocation as
+ * the given block could be the keeper block.
+ */
+ context->isReset = false;
+
/*
* If there is enough room in the active allocation block, we will put the
* chunk into that block. Else must start a new one.
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 4b4155b863..5a351c26df 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -352,6 +352,8 @@ GenerationAlloc(MemoryContext context, Size size, int flags)
Size chunk_size;
Size required_size;
+ context->isReset = false;
+
Assert(GenerationIsValid(set));
#ifdef MEMORY_CONTEXT_CHECKING
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index c214d323c8..01f3bec028 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1070,8 +1070,6 @@ MemoryContextAlloc(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1084,6 +1082,8 @@ MemoryContextAlloc(MemoryContext context, Size size)
*/
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
@@ -1104,10 +1104,10 @@ MemoryContextAllocZero(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
/*
@@ -1135,9 +1135,10 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
AllocSizeIsValid(size)))
elog(ERROR, "invalid memory alloc request size %zu", size);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, flags);
+
+ Assert(!context->isReset);
+
if (unlikely(ret == NULL))
return NULL;
@@ -1211,8 +1212,6 @@ palloc(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1224,6 +1223,9 @@ palloc(Size size)
* function instead.
*/
ret = context->methods->alloc(context, size, 0);
+
+ Assert(!context->isReset);
+
/* We expect OOM to be handled by the alloc function */
Assert(ret != NULL);
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
@@ -1241,10 +1243,10 @@ palloc0(Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, 0);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
MemSetAligned(ret, 0, size);
@@ -1262,9 +1264,10 @@ palloc_extended(Size size, int flags)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
ret = context->methods->alloc(context, size, flags);
+
+ Assert(!context->isReset);
+
if (unlikely(ret == NULL))
{
return NULL;
@@ -1532,8 +1535,6 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
Assert(MemoryContextIsValid(context));
AssertNotInCriticalSection(context);
- context->isReset = false;
-
/*
* For efficiency reasons, we purposefully offload the handling of
* allocation failures to the MemoryContextMethods implementation as this
@@ -1546,6 +1547,8 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
*/
ret = context->methods->alloc(context, size, MCXT_ALLOC_HUGE);
+ Assert(!context->isReset);
+
VALGRIND_MEMPOOL_ALLOC(context, ret, size);
return ret;
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index bc91446cb3..1160aa7e20 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -533,6 +533,12 @@ SlabAlloc(MemoryContext context, Size size, int flags)
{
dlist_node *node = dclist_pop_head_node(&slab->emptyblocks);
+ /*
+ * We should already have already un-reset the context if we've
+ * got a block on the emptyblocks list.
+ */
+ Assert(!context->isReset);
+
block = dlist_container(SlabBlock, node, node);
/*
@@ -546,6 +552,8 @@ SlabAlloc(MemoryContext context, Size size, int flags)
}
else
{
+ context->isReset = false;
+
block = (SlabBlock *) malloc(slab->blockSize);
if (unlikely(block == NULL))
@@ -581,6 +589,12 @@ SlabAlloc(MemoryContext context, Size size, int flags)
Assert(!dlist_is_empty(blocklist));
+ /*
+ * We should already have already un-reset the context if we've got a
+ * non-zero curBlocklistIndex
+ */
+ Assert(!context->isReset);
+
/* grab the block from the blocklist */
block = dlist_head_element(SlabBlock, node, blocklist);
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index edc0257f36..72539cfc5d 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -61,7 +61,10 @@ typedef struct MemoryContextMethods
* Function to handle memory allocation requests of 'size' to allocate
* memory into the given 'context'. The function must handle flags
* MCXT_ALLOC_HUGE and MCXT_ALLOC_NO_OOM. MCXT_ALLOC_ZERO is handled by
- * the calling function.
+ * the calling function. The implementation must handle setting
+ * MemoryContext.isReset to false before returning. Leaving this up to
+ * the implementing function can allow this flag to be unset more
+ * efficiently.
*/
void *(*alloc) (MemoryContext context, Size size, int flags);
--
2.40.1.windows.1
[text/plain] Function-to-test-palloc-pfree-performance.patch.txt (8.2K, ../CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com/5-Function-to-test-palloc-pfree-performance.patch.txt)
download | inline diff:
From 519ba53c785e1858952c2fa2692dc34aed54b76a Mon Sep 17 00:00:00 2001
From: David Rowley <[email protected]>
Date: Wed, 25 May 2022 14:24:37 +1200
Subject: [PATCH v99] Function to test palloc/pfree performance
---
src/backend/utils/adt/mcxtfuncs.c | 213 ++++++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 12 ++
2 files changed, 225 insertions(+)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 4708d73f5f..416235b473 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,6 +15,8 @@
#include "postgres.h"
+#include <time.h>
+
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
@@ -193,3 +195,214 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+typedef struct AllocateTestNext
+{
+ struct AllocateTestNext *next; /* ptr to the next allocation */
+} AllocateTestNext;
+
+/* #define ALLOCATE_TEST_DEBUG */
+/*
+ * pg_allocate_memory_test
+ * Used to test the performance of a memory context types
+ */
+Datum
+pg_allocate_memory_test(PG_FUNCTION_ARGS)
+{
+ int32 chunk_size = PG_GETARG_INT32(0);
+ int64 keep_memory = PG_GETARG_INT64(1);
+ int64 total_alloc = PG_GETARG_INT64(2);
+ text *context_type_text = PG_GETARG_TEXT_PP(3);
+ char *context_type;
+ int64 curr_memory_use = 0;
+ int64 remaining_alloc_bytes = total_alloc;
+ MemoryContext context;
+ MemoryContext oldContext;
+ AllocateTestNext *next_free_ptr = NULL;
+ AllocateTestNext *last_alloc = NULL;
+ clock_t start, end;
+
+ if (chunk_size < sizeof(AllocateTestNext))
+ elog(ERROR, "chunk_size (%d) must be at least %ld bytes", chunk_size,
+ sizeof(AllocateTestNext));
+ if (keep_memory > total_alloc)
+ elog(ERROR, "keep_memory (" INT64_FORMAT ") must be less than total_alloc (" INT64_FORMAT ")",
+ keep_memory, total_alloc);
+
+ context_type = text_to_cstring(context_type_text);
+
+ start = clock();
+
+ if (strcmp(context_type, "generation") == 0)
+ context = GenerationContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "aset") == 0)
+ context = AllocSetContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "slab") == 0)
+ context = SlabContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_MAXSIZE,
+ chunk_size);
+ else
+ elog(ERROR, "context_type must be \"generation\", \"aset\" or \"slab\"");
+
+ oldContext = MemoryContextSwitchTo(context);
+
+ while (remaining_alloc_bytes > 0)
+ {
+ AllocateTestNext *curr_alloc;
+
+ CHECK_FOR_INTERRUPTS();
+
+ /* Allocate the memory and update the counters */
+ curr_alloc = (AllocateTestNext *) palloc(chunk_size);
+ remaining_alloc_bytes -= chunk_size;
+ curr_memory_use += chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "alloc %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", curr_alloc, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ /*
+ * Point the last allocate to this one so that we can free allocations
+ * starting with the oldest first.
+ */
+ curr_alloc->next = NULL;
+ if (last_alloc != NULL)
+ last_alloc->next = curr_alloc;
+
+ if (next_free_ptr == NULL)
+ {
+ /*
+ * Remember the first chunk to free. We will follow the ->next
+ * pointers to find the next chunk to free when freeing memory
+ */
+ next_free_ptr = curr_alloc;
+ }
+
+ /*
+ * If the currently allocated memory has reached or exceeded the amount
+ * of memory we want to keep allocated at once then we'd better free
+ * some. Since all allocations are the same size we only need to free
+ * one allocation per loop.
+ */
+ if (curr_memory_use >= keep_memory)
+ {
+ AllocateTestNext *next = next_free_ptr->next;
+
+ /* free the memory and update the current memory usage */
+ pfree(next_free_ptr);
+ curr_memory_use -= chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "free %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", next_free_ptr, curr_memory_use, remaining_alloc_bytes);
+#endif
+ /* get the next chunk to free */
+ next_free_ptr = next;
+ }
+
+ if (curr_memory_use > 0)
+ last_alloc = curr_alloc;
+ else
+ last_alloc = NULL;
+ }
+
+ /* cleanup loop -- pfree remaining memory */
+ while (next_free_ptr != NULL)
+ {
+ AllocateTestNext *next = next_free_ptr->next;
+
+ /* free the memory and update the current memory usage */
+ pfree(next_free_ptr);
+ curr_memory_use -= chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "free %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", next_free_ptr, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ next_free_ptr = next;
+ }
+
+ MemoryContextSwitchTo(oldContext);
+
+ end = clock();
+
+ PG_RETURN_FLOAT8((double) (end - start) / CLOCKS_PER_SEC);
+}
+
+Datum
+pg_allocate_memory_test_reset(PG_FUNCTION_ARGS)
+{
+ int32 chunk_size = PG_GETARG_INT32(0);
+ int64 keep_memory = PG_GETARG_INT64(1);
+ int64 total_alloc = PG_GETARG_INT64(2);
+ text *context_type_text = PG_GETARG_TEXT_PP(3);
+ char *context_type;
+ int64 curr_memory_use = 0;
+ int64 remaining_alloc_bytes = total_alloc;
+ MemoryContext context;
+ MemoryContext oldContext;
+ clock_t start, end;
+
+ if (chunk_size < 1)
+ elog(ERROR, "size of chunk must be above 0");
+ if (keep_memory > total_alloc)
+ elog(ERROR, "keep_memory (" INT64_FORMAT ") must be less than total_alloc (" INT64_FORMAT ")",
+ keep_memory, total_alloc);
+
+ context_type = text_to_cstring(context_type_text);
+
+ start = clock();
+
+ if (strcmp(context_type, "generation") == 0)
+ context = GenerationContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "aset") == 0)
+ context = AllocSetContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_SIZES);
+ else if (strcmp(context_type, "slab") == 0)
+ context = SlabContextCreate(CurrentMemoryContext,
+ "pg_allocate_memory_test",
+ ALLOCSET_DEFAULT_MAXSIZE,
+ chunk_size);
+ else
+ elog(ERROR, "context_type must be \"generation\", \"aset\" or \"slab\"");
+
+ oldContext = MemoryContextSwitchTo(context);
+
+ while (remaining_alloc_bytes > 0)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ /* Allocate the memory and update the counters */
+ (void) palloc(chunk_size);
+ remaining_alloc_bytes -= chunk_size;
+ curr_memory_use += chunk_size;
+
+#ifdef ALLOCATE_TEST_DEBUG
+ elog(NOTICE, "alloc %p (curr_memory_use " INT64_FORMAT " bytes, remaining_alloc_bytes " INT64_FORMAT ")", curr_alloc, curr_memory_use, remaining_alloc_bytes);
+#endif
+
+ /*
+ * If the currently allocated memory has reached or exceeded the amount
+ * of memory we want to keep allocated at once then reset the context.
+ */
+ if (curr_memory_use >= keep_memory)
+ {
+ curr_memory_use = 0;
+ MemoryContextReset(context);
+ }
+ }
+
+ MemoryContextSwitchTo(oldContext);
+ MemoryContextDelete(context);
+
+ end = clock();
+
+ PG_RETURN_FLOAT8((double) (end - start) / CLOCKS_PER_SEC);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 9c120fc2b7..f307229c60 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8274,6 +8274,18 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# just for testing memory context allocation speed
+{ oid => '9319', descr => 'for testing performance of allocation and freeing',
+ proname => 'pg_allocate_memory_test', provolatile => 'v',
+ prorettype => 'float8', proargtypes => 'int4 int8 int8 text',
+ prosrc => 'pg_allocate_memory_test' },
+
+# just for testing memory context allocation speed
+{ oid => '9320', descr => 'for testing performance of allocation and resetting',
+ proname => 'pg_allocate_memory_test_reset', provolatile => 'v',
+ prorettype => 'float8', proargtypes => 'int4 int8 int8 text',
+ prosrc => 'pg_allocate_memory_test_reset' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
--
2.40.1.windows.1
[image/png] benchmark_results.png (117.3K, ../CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com/6-benchmark_results.png)
download | view image
view thread (13+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Avoid stack frame setup in performance critical routines using tail calls
In-Reply-To: <CAApHDvqss7-a9c51nj+f9xyAr15wjLB6teHsxPe-NwLCNqiJbg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox