public inbox for [email protected]
help / color / mirror / Atom feedFrom: Anton A. Melnikov <[email protected]>
To: Tomas Vondra <[email protected]>
To: James Hunter <[email protected]>
To: Tomas Vondra <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: Andrei Lepikhov <[email protected]>
Cc: Stephen Frost <[email protected]>
Cc: [email protected]
Cc: Arne Roland <[email protected]>
Cc: [email protected] <[email protected]>
Cc: vignesh C <[email protected]>
Cc: Justin Pryzby <[email protected]>
Cc: Ibrar Ahmed <[email protected]>
Cc: stephen.frost <[email protected]>
Subject: Re: Add the ability to limit the amount of memory that can be allocated to backends.
Date: Sat, 28 Dec 2024 15:36:37 +0300
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<CAJVSvF6NRsnAqOYSkveiBERU+QyHV1vBrjd0ORFD+kw2N5-rEQ@mail.gmail.com>
<[email protected]>
Hi!
On 28.12.2024 04:48, Tomas Vondra wrote:
> On 12/27/24 20:14, James Hunter wrote:
>> Reviving this thread, because I am thinking about something related --
>> please ignore the "On Fri, Dec 27, 2024" date, this seems to be an
>> artifact of me re-sending the message, from the list archive. The
>> original message was from January 28, 2024.
>>
>> On Fri, Dec 27, 2024 at 11:02 AM Tomas Vondra
>> <[email protected]> wrote:
>>>
>>> Firstly, I agree with the goal of having a way to account for memory
>>> used by the backends, and also ability to enforce some sort of limit.
>>> It's difficult to track the memory at the OS level (interpreting RSS
>>> values is not trivial), and work_mem is not sufficient to enforce a
>>> backend-level limit, not even talking about a global limit.
>>>
>>> But as I said earlier, it seems quite strange to start by introducing
>>> some sort of global limit, combining memory for all backends. I do
>>> understand that the intent is to have such global limit in order to
>>> prevent issues with the OOM killer and/or not to interfere with other
>>> stuff running on the same machine. And while I'm not saying we should
>>> not have such limit, every time I wished to have a memory limit it was a
>>> backend-level one. Ideally a workmem-like limit that would "adjust" the
>>> work_mem values used by the optimizer (but that's not what this patch
>>> aims to do), or at least a backstop in case something goes wrong (say, a
>>> memory leak, OLTP application issuing complex queries, etc.).
>>
>> I think what Tomas suggests is the right strategy.
I'm also interested in this topic. And agreed that it's best to move from the limit
for a separate backend to the global one. In more details let me suggest
the following steps or parts:
1) realize memory limitation for a separate backend independent from the work_mem GUC;
2) add workmem-like limit that would "adjust" the work_mem values used by
the optimize as Thomas suggested;
3) add global limit for all backends.
As for p.1 there is a patch that was originally suggested by my colleague
Maxim Orlov <[email protected]> and which i modified for the current master.
This patch introduces the only max_backend_memory GUC that specifies
the maximum amount of memory that can be allocated to a backend.
Zero value means no limit.
If the allocated memory size is exceeded, a standard "out of memory" error will be issued.
Also the patch introduces the pg_get_backend_memory_contexts_total_bytes() function,
which allows to know how many bytes have already been allocated
to the process in contexts. And in the build with asserts it adds
the pg_get_backend_memory_allocation_stats() function that allows
to get additional information about memory allocations for debug purposes.
>> This strategy solves the ongoing problem of how to set work_mem, if
>> some queries have lots of operators and others don't -- now we just
>> set backend_work_mem, as a limit on the entire query's total work_mem.
>> And a bit of integration with the optimizer will allow us to
>> distribute the total backend_work_mem to individual execution nodes,
>> with the goal of minimizing spilling, without exceeding the
>> backend_work_mem limit.
As for p.2 maybe one can set a maximum number of parallel sort or
hash table operations before writing to disk instead of absolute
value in the work_mem GUC? E.g. introduce а max_query_operations GUC
or a variable in such a way that old work_mem will be equal
to max_backend_memory divided by max_query_operations.
What do you think about such approach?
With the best wishes,
--
Anton A. Melnikov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
Attachments:
[text/x-patch] 0001-v1-limit-backend-heap-memory-allocation.patch (27.0K, ../[email protected]/2-0001-v1-limit-backend-heap-memory-allocation.patch)
download | inline diff:
From 8ec3548604bf6a1f5f4c290495843e7080d09660 Mon Sep 17 00:00:00 2001
From: "Anton A. Melnikov" <[email protected]>
Date: Sat, 28 Dec 2024 12:47:24 +0300
Subject: [PATCH] Limit backend heap memory allocation
This per-backend GUC can limit the amount of heap allocated
memory. Upon reaching this limit "out of memory" error will
be triggered.
GUC is diabled by default.
Authors: Maxim Orlov <[email protected]>, Anton A. Melnikov <[email protected]>
---
doc/src/sgml/config.sgml | 17 +
src/backend/utils/activity/backend_status.c | 335 ++++++++++++++++++
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/backend/utils/mmgr/aset.c | 36 +-
src/backend/utils/mmgr/generation.c | 18 +-
src/backend/utils/mmgr/slab.c | 17 +-
src/include/catalog/pg_proc.dat | 14 +
src/include/utils/backend_status.h | 10 +
src/test/modules/Makefile | 4 +
.../modules/test_backend_memory/.gitignore | 4 +
src/test/modules/test_backend_memory/Makefile | 14 +
src/test/modules/test_backend_memory/README | 1 +
.../t/001_max_backend_memory.pl | 85 +++++
14 files changed, 540 insertions(+), 26 deletions(-)
create mode 100644 src/test/modules/test_backend_memory/.gitignore
create mode 100644 src/test/modules/test_backend_memory/Makefile
create mode 100644 src/test/modules/test_backend_memory/README
create mode 100644 src/test/modules/test_backend_memory/t/001_max_backend_memory.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fbdd6ce574..4099d6e1a8 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -2337,6 +2337,23 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-backend-memory" xreflabel="max_backend_memory">
+ <term><varname>max_backend_memory</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_backend_memory</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory that can be allocated to a backend.
+ Zero means no limit.
+ </para>
+ <para>
+ Default: 0
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/backend/utils/activity/backend_status.c b/src/backend/utils/activity/backend_status.c
index bf33e33a4e..a41f004cf7 100644
--- a/src/backend/utils/activity/backend_status.c
+++ b/src/backend/utils/activity/backend_status.c
@@ -12,6 +12,7 @@
#include "postgres.h"
#include "access/xact.h"
+#include "funcapi.h"
#include "libpq/libpq-be.h"
#include "miscadmin.h"
#include "pg_trace.h"
@@ -20,6 +21,7 @@
#include "storage/proc.h" /* for MyProc */
#include "storage/procarray.h"
#include "utils/ascii.h"
+#include "utils/builtins.h"
#include "utils/guc.h" /* for application_name */
#include "utils/memutils.h"
@@ -73,6 +75,17 @@ static void pgstat_beshutdown_hook(int code, Datum arg);
static void pgstat_read_current_status(void);
static void pgstat_setup_backend_status_context(void);
+/*
+ * Total memory size allocated by backend.
+ */
+static Size my_allocated_bytes = 0;
+
+/*
+ * GUC variable
+ *
+ * Max backend memory allocation allowed (MB). 0 = disabled
+ */
+int max_backend_memory_size_mb = 0;
/*
* Report shared-memory space needed by BackendStatusShmemInit.
@@ -1222,3 +1235,325 @@ pgstat_clip_activity(const char *raw_activity)
return activity;
}
+
+#ifdef USE_ASSERT_CHECKING
+/*
+ * Single memory allocation.
+ */
+typedef struct
+{
+ void *addr;
+ Size size;
+ bool deleted;
+} HeapBlock;
+
+/*
+ * Dynamic array of all allocated memory.
+ */
+typedef struct
+{
+ HeapBlock *blocks;
+ Size size;
+ Size capacity;
+ Size ndeleted;
+} HeapBlockStat;
+
+static HeapBlockStat heapstat =
+{
+ .blocks = NULL,
+ .size = 0,
+ .capacity = 0,
+ .ndeleted = 0
+};
+
+/*
+ * Extend heapstat allocation stat array.
+ */
+static inline void
+HeapBlockStatExtend(void)
+{
+ if (heapstat.capacity == 0)
+ {
+#define INIT_MEMORY_CAPACITY 1024
+ heapstat.capacity = INIT_MEMORY_CAPACITY;
+ heapstat.blocks = (HeapBlock *) malloc(heapstat.capacity *
+ sizeof(heapstat.blocks[0]));
+ }
+ else if (heapstat.capacity == heapstat.size)
+ {
+ heapstat.capacity *= 2;
+ heapstat.blocks = (HeapBlock *) realloc(heapstat.blocks,
+ heapstat.capacity *
+ sizeof(heapstat.blocks[0]));
+ }
+}
+
+/*
+ * Compare HeapBlocks. Put not deleted in the beginning of an array.
+ */
+static int
+HeapBlockCmp(const void *p, const void *q)
+{
+ const HeapBlock *a = (const HeapBlock *) p,
+ *b = (const HeapBlock *) q;
+ int arg1 = a->deleted,
+ arg2 = b->deleted;
+
+ if (arg1 < arg2)
+ return -1;
+ if (arg1 > arg2)
+ return 1;
+ return 0;
+}
+
+/*
+ * Put all not deleted items in the beginning.
+ */
+static void
+HeapBlockStatCompactify(void)
+{
+ if (heapstat.ndeleted < heapstat.size / 2)
+ return;
+
+ qsort(heapstat.blocks, heapstat.size, sizeof(heapstat.blocks[0]), HeapBlockCmp);
+ heapstat.size -= heapstat.ndeleted;
+ heapstat.ndeleted = 0;
+}
+
+/*
+ * Push new memory allocation.
+ */
+static void
+HeapBlockStatPush(void *ptr, Size size)
+{
+ Size i;
+
+ HeapBlockStatExtend();
+ HeapBlockStatCompactify();
+
+ for (i = 0; i < heapstat.size; ++i)
+ Assert(heapstat.blocks[i].deleted || heapstat.blocks[i].addr != ptr);
+
+ Assert(heapstat.ndeleted <= heapstat.size);
+
+ /* Try to find deleted item to reuse it... */
+ for (i = 0; i < heapstat.size; ++i)
+ {
+ if (heapstat.blocks[i].deleted == true)
+ {
+ heapstat.blocks[i].addr = ptr;
+ heapstat.blocks[i].size = size;
+ heapstat.blocks[i].deleted = false;
+ --heapstat.ndeleted;
+ Assert(heapstat.ndeleted <= heapstat.size);
+ return;
+ }
+ }
+
+ /* ... no empty places, append! */
+ heapstat.blocks[i].addr = ptr;
+ heapstat.blocks[i].size = size;
+ heapstat.blocks[i].deleted = false;
+
+ ++heapstat.size;
+}
+
+/*
+ * Pop memory allocation.
+ */
+static Size
+HeapBlockStatPop(void *ptr)
+{
+ Size i;
+
+ for (i = 0; i < heapstat.size; ++i)
+ {
+ if (heapstat.blocks[i].addr == ptr)
+ {
+ Assert(heapstat.blocks[i].deleted == false);
+ heapstat.blocks[i].deleted = true;
+ ++heapstat.ndeleted;
+ Assert(heapstat.ndeleted <= heapstat.size);
+ return heapstat.blocks[i].size;
+ }
+ }
+
+ /* should not get here... */
+ Assert(i != heapstat.size);
+ return 0;
+}
+#endif /* USE_ASSERT_CHECKING */
+
+/*
+ * Count backend allocated memory.
+ * If limit is set (max_backend_memory_size_mb) and check=true, then need to
+ * check my_allocated_bytes for going beyond the limit.
+ */
+static bool
+pgstat_alloc(Size size, bool check)
+{
+ /* Exclude auxiliary processes from the check */
+ switch (MyBackendType)
+ {
+ case B_STARTUP:
+ case B_ARCHIVER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_WAL_WRITER:
+ case B_WAL_RECEIVER:
+ case B_WAL_SUMMARIZER:
+ return true;
+ default:
+ break;
+ }
+
+ my_allocated_bytes += size;
+
+ if (max_backend_memory_size_mb == 0 || !check)
+ return true;
+
+ /* Check for going beyond the limit */
+#define TO_BYTES(mb) ((Size)(mb) * 1024 * 1024)
+ if (my_allocated_bytes > TO_BYTES(max_backend_memory_size_mb))
+ {
+ my_allocated_bytes -= size;
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Count deallocated backend memory.
+ */
+static void
+pgstat_free(Size size)
+{
+ /* Exclude auxiliary processes from the check */
+ switch (MyBackendType)
+ {
+ case B_STARTUP:
+ case B_ARCHIVER:
+ case B_BG_WRITER:
+ case B_CHECKPOINTER:
+ case B_WAL_WRITER:
+ case B_WAL_RECEIVER:
+ case B_WAL_SUMMARIZER:
+ return;
+ default:
+ break;
+ }
+
+ Assert(my_allocated_bytes >= size);
+ my_allocated_bytes -= size;
+}
+
+/*
+ * malloc() with bytes counter
+ */
+void *
+malloc_and_count(Size size)
+{
+ void *ptr;
+
+ if (!pgstat_alloc(size, true))
+ return NULL;
+
+ ptr = malloc(size);
+ if (ptr == NULL)
+ pgstat_free(size);
+#ifdef USE_ASSERT_CHECKING
+ else
+ HeapBlockStatPush(ptr, size);
+#endif
+
+ return ptr;
+}
+
+/*
+ * realloc() with bytes counter
+ */
+void *
+realloc_and_count(void *ptr, Size new_size, Size old_size)
+{
+ Assert(old_size == HeapBlockStatPop(ptr));
+
+ pgstat_free(old_size);
+
+ if (!pgstat_alloc(new_size, true))
+ {
+ /* New buffer can not be alloced, so need to restore old one */
+ pgstat_alloc(old_size, false);
+#ifdef USE_ASSERT_CHECKING
+ HeapBlockStatPush(ptr, old_size);
+#endif
+ return NULL;
+ }
+
+ ptr = realloc(ptr, new_size);
+ if (ptr == NULL)
+ pgstat_free(new_size);
+#ifdef USE_ASSERT_CHECKING
+ else
+ HeapBlockStatPush(ptr, new_size);
+#endif
+
+ return ptr;
+}
+
+/*
+ * free() with bytes counter
+ */
+void
+free_and_count(void *ptr, Size size)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(size == HeapBlockStatPop(ptr));
+#endif
+
+ pgstat_free(size);
+ free(ptr);
+}
+
+/*
+ * pg_get_backend_memory_contexts_total_bytes
+ * Total amount of bytes in all allocated contexts.
+ */
+Datum
+pg_get_backend_memory_contexts_total_bytes(PG_FUNCTION_ARGS)
+{
+ PG_RETURN_INT64(my_allocated_bytes);
+}
+
+/*
+ * pg_get_backend_memory_allocation_stats
+ * SQL SRF showing backend allocated memory.
+ */
+Datum
+pg_get_backend_memory_allocation_stats(PG_FUNCTION_ARGS)
+{
+#ifdef USE_ASSERT_CHECKING
+#define MEMORY_ALLOCATION_COLS 3
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ Datum values[MEMORY_ALLOCATION_COLS];
+ bool nulls[MEMORY_ALLOCATION_COLS];
+ Size i;
+ char text[16];
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ memset(nulls, 0, sizeof(nulls));
+ for (i = 0; i < heapstat.size; ++i)
+ {
+ pg_snprintf(text, sizeof(text), "%p", heapstat.blocks[i].addr);
+ values[0] = CStringGetTextDatum(text);
+ values[1] = Int64GetDatum(heapstat.blocks[i].size);
+ values[2] = BoolGetDatum(heapstat.blocks[i].deleted);
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
+ }
+#else
+ elog(ERROR, "this only works for builds with asserts enabled");
+#endif
+
+ return (Datum) 0;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8cf1afbad2..f888d417de 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3732,6 +3732,16 @@ struct config_int ConfigureNamesInt[] =
SCRAM_SHA_256_DEFAULT_ITERATIONS, 1, INT_MAX,
NULL, NULL, NULL
},
+ {
+ {"max_backend_memory", PGC_SU_BACKEND, RESOURCES_MEM,
+ gettext_noop("Restrict total backend memory allocations to this max (rounded up to the nearest MB)."),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &max_backend_memory_size_mb,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
/* End-of-list marker */
{
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a2ac7575ca..1854601ca9 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -171,6 +171,7 @@
#serializable_buffers = 32 # memory for pg_serial
#subtransaction_buffers = 0 # memory for pg_subtrans (0 = auto)
#transaction_buffers = 0 # memory for pg_xact (0 = auto)
+#max_backend_memory = 0 # limit amount of per-backend allocated memory in megabytes
# - Disk -
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index dede30dd86..e304da7c3c 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -47,6 +47,7 @@
#include "postgres.h"
#include "port/pg_bitutils.h"
+#include "utils/backend_status.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -441,7 +442,7 @@ AllocSetContextCreateInternal(MemoryContext parent,
* Allocate the initial block. Unlike other aset.c blocks, it starts with
* the context header and its block header follows that.
*/
- set = (AllocSet) malloc(firstBlockSize);
+ set = (AllocSet) malloc_and_count(firstBlockSize);
if (set == NULL)
{
if (TopMemoryContext)
@@ -579,13 +580,16 @@ AllocSetReset(MemoryContext context)
}
else
{
+ Size deallocation_size;
+
/* Normal case, release the block */
- context->mem_allocated -= block->endptr - ((char *) block);
+ deallocation_size = block->endptr - ((char *) block);
+ context->mem_allocated -= deallocation_size;
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, block->freeptr - ((char *) block));
#endif
- free(block);
+ free_and_count(block, deallocation_size);
}
block = next;
}
@@ -649,7 +653,7 @@ AllocSetDelete(MemoryContext context)
freelist->num_free--;
/* All that remains is to free the header/initial block */
- free(oldset);
+ free_and_count(oldset, oldset->header.mem_allocated);
}
Assert(freelist->num_free == 0);
}
@@ -666,16 +670,20 @@ AllocSetDelete(MemoryContext context)
while (block != NULL)
{
AllocBlock next = block->next;
+ Size deallocation_size = 0;
if (!IsKeeperBlock(set, block))
- context->mem_allocated -= block->endptr - ((char *) block);
+ {
+ deallocation_size = block->endptr - ((char *) block);
+ context->mem_allocated -= deallocation_size;
+ }
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, block->freeptr - ((char *) block));
#endif
if (!IsKeeperBlock(set, block))
- free(block);
+ free_and_count(block, deallocation_size);
block = next;
}
@@ -683,7 +691,7 @@ AllocSetDelete(MemoryContext context)
Assert(context->mem_allocated == keepersize);
/* Finally, free the context header, including the keeper block */
- free(set);
+ free_and_count(set, keepersize);
}
/*
@@ -712,7 +720,7 @@ AllocSetAllocLarge(MemoryContext context, Size size, int flags)
#endif
blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
- block = (AllocBlock) malloc(blksize);
+ block = (AllocBlock) malloc_and_count(blksize);
if (block == NULL)
return MemoryContextAllocationFailure(context, size, flags);
@@ -905,7 +913,7 @@ AllocSetAllocFromNewBlock(MemoryContext context, Size size, int flags,
blksize <<= 1;
/* Try to allocate it */
- block = (AllocBlock) malloc(blksize);
+ block = (AllocBlock) malloc_and_count(blksize);
/*
* We could be asking for pretty big blocks here, so cope if malloc fails.
@@ -916,7 +924,7 @@ AllocSetAllocFromNewBlock(MemoryContext context, Size size, int flags,
blksize >>= 1;
if (blksize < required_size)
break;
- block = (AllocBlock) malloc(blksize);
+ block = (AllocBlock) malloc_and_count(blksize);
}
if (block == NULL)
@@ -1071,6 +1079,7 @@ AllocSetFree(void *pointer)
{
/* Release single-chunk block. */
AllocBlock block = ExternalChunkGetBlock(chunk);
+ Size deallocation_size;
/*
* Try to verify that we have a sane block pointer: the block header
@@ -1099,12 +1108,13 @@ AllocSetFree(void *pointer)
if (block->next)
block->next->prev = block->prev;
- set->header.mem_allocated -= block->endptr - ((char *) block);
+ deallocation_size = block->endptr - ((char *) block);
+ set->header.mem_allocated -= deallocation_size;
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, block->freeptr - ((char *) block));
#endif
- free(block);
+ free_and_count(block, deallocation_size);
}
else
{
@@ -1223,7 +1233,7 @@ AllocSetRealloc(void *pointer, Size size, int flags)
blksize = chksize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
oldblksize = block->endptr - ((char *) block);
- block = (AllocBlock) realloc(block, blksize);
+ block = (AllocBlock) realloc_and_count(block, blksize, oldblksize);
if (block == NULL)
{
/* Disallow access to the chunk header. */
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 0238c111d2..c7ec138110 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -37,6 +37,7 @@
#include "lib/ilist.h"
#include "port/pg_bitutils.h"
+#include "utils/backend_status.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -206,7 +207,7 @@ GenerationContextCreate(MemoryContext parent,
* Allocate the initial block. Unlike other generation.c blocks, it
* starts with the context header and its block header follows that.
*/
- set = (GenerationContext *) malloc(allocSize);
+ set = (GenerationContext *) malloc_and_count(allocSize);
if (set == NULL)
{
MemoryContextStats(TopMemoryContext);
@@ -330,7 +331,7 @@ GenerationDelete(MemoryContext context)
/* Reset to release all releasable GenerationBlocks */
GenerationReset(context);
/* And free the context header and keeper block */
- free(context);
+ free_and_count(context, context->mem_allocated + MAXALIGN(sizeof(GenerationContext)));
}
/*
@@ -361,7 +362,7 @@ GenerationAllocLarge(MemoryContext context, Size size, int flags)
required_size = chunk_size + Generation_CHUNKHDRSZ;
blksize = required_size + Generation_BLOCKHDRSZ;
- block = (GenerationBlock *) malloc(blksize);
+ block = (GenerationBlock *) malloc_and_count(blksize);
if (block == NULL)
return MemoryContextAllocationFailure(context, size, flags);
@@ -482,7 +483,7 @@ GenerationAllocFromNewBlock(MemoryContext context, Size size, int flags,
if (blksize < required_size)
blksize = pg_nextpower2_size_t(required_size);
- block = (GenerationBlock *) malloc(blksize);
+ block = (GenerationBlock *) malloc_and_count(blksize);
if (block == NULL)
return MemoryContextAllocationFailure(context, size, flags);
@@ -663,6 +664,9 @@ GenerationBlockFreeBytes(GenerationBlock *block)
static inline void
GenerationBlockFree(GenerationContext *set, GenerationBlock *block)
{
+ /* Have to store the value locally, block will be wiped. */
+ Size blksize = block->blksize;
+
/* Make sure nobody tries to free the keeper block */
Assert(!IsKeeperBlock(set, block));
/* We shouldn't be freeing the freeblock either */
@@ -671,13 +675,13 @@ GenerationBlockFree(GenerationContext *set, GenerationBlock *block)
/* release the block from the list of blocks */
dlist_delete(&block->node);
- ((MemoryContext) set)->mem_allocated -= block->blksize;
+ ((MemoryContext) set)->mem_allocated -= blksize;
#ifdef CLOBBER_FREED_MEMORY
- wipe_mem(block, block->blksize);
+ wipe_mem(block, blksize);
#endif
- free(block);
+ free_and_count(block, blksize);
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 3e15d59683..b8baea6c98 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -69,6 +69,7 @@
#include "postgres.h"
#include "lib/ilist.h"
+#include "utils/backend_status.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -361,7 +362,7 @@ SlabContextCreate(MemoryContext parent,
- slab = (SlabContext *) malloc(Slab_CONTEXT_HDRSZ(chunksPerBlock));
+ slab = (SlabContext *) malloc_and_count(Slab_CONTEXT_HDRSZ(chunksPerBlock));
if (slab == NULL)
{
MemoryContextStats(TopMemoryContext);
@@ -451,7 +452,7 @@ SlabReset(MemoryContext context)
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, slab->blockSize);
#endif
- free(block);
+ free_and_count(block, slab->blockSize);
context->mem_allocated -= slab->blockSize;
}
@@ -467,7 +468,7 @@ SlabReset(MemoryContext context)
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, slab->blockSize);
#endif
- free(block);
+ free_and_count(block, slab->blockSize);
context->mem_allocated -= slab->blockSize;
}
}
@@ -484,10 +485,14 @@ SlabReset(MemoryContext context)
void
SlabDelete(MemoryContext context)
{
+#ifdef MEMORY_CONTEXT_CHECKING
+ SlabContext *slab = (SlabContext *) context;
+#endif
+
/* Reset to release all the SlabBlocks */
SlabReset(context);
/* And free the context header */
- free(context);
+ free_and_count(context, Slab_CONTEXT_HDRSZ(slab->chunksPerBlock));
}
/*
@@ -562,7 +567,7 @@ SlabAllocFromNewBlock(MemoryContext context, Size size, int flags)
}
else
{
- block = (SlabBlock *) malloc(slab->blockSize);
+ block = (SlabBlock *) malloc_and_count(slab->blockSize);
if (unlikely(block == NULL))
return MemoryContextAllocationFailure(context, size, flags);
@@ -795,7 +800,7 @@ SlabFree(void *pointer)
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, slab->blockSize);
#endif
- free(block);
+ free_and_count(block, slab->blockSize);
slab->header.mem_allocated -= slab->blockSize;
}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2dcc2d42da..16dd4a0d97 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8442,6 +8442,20 @@
proargmodes => '{o,o,o,o,o,o,o,o,o,o}',
proargnames => '{name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+{ oid => '8085',
+ descr => 'get the allocated memory in all memory contexts of local backend',
+ proname => 'pg_get_backend_memory_contexts_total_bytes', provolatile => 'v',
+ proparallel => 'r', prorettype => 'int8', proargtypes => '',
+ prosrc => 'pg_get_backend_memory_contexts_total_bytes' },
+{ oid => '8005',
+ descr => 'information about memory allocation of local backend',
+ proname => 'pg_get_backend_memory_allocation_stats', prorows => '1024',
+ proretset => 't', provolatile => 'v', proparallel => 'r',
+ prorettype => 'record', proargtypes => '',
+ proallargtypes => '{text,int8,bool}',
+ proargmodes => '{o,o,o}',
+ proargnames => '{pointer, size, deleted}',
+ prosrc => 'pg_get_backend_memory_allocation_stats' },
# logging memory contexts of the specified backend
{ oid => '4543', descr => 'log memory contexts of the specified backend',
diff --git a/src/include/utils/backend_status.h b/src/include/utils/backend_status.h
index 4e8b39a66d..26b9338b2f 100644
--- a/src/include/utils/backend_status.h
+++ b/src/include/utils/backend_status.h
@@ -11,6 +11,7 @@
#define BACKEND_STATUS_H
#include "datatype/timestamp.h"
+#include "fmgr.h"
#include "libpq/pqcomm.h"
#include "miscadmin.h" /* for BackendType */
#include "storage/procnumber.h"
@@ -285,6 +286,7 @@ typedef struct LocalPgBackendStatus
*/
extern PGDLLIMPORT bool pgstat_track_activities;
extern PGDLLIMPORT int pgstat_track_activity_query_size;
+extern PGDLLIMPORT int max_backend_memory_size_mb;
/* ----------
@@ -337,5 +339,13 @@ extern LocalPgBackendStatus *pgstat_get_local_beentry_by_proc_number(ProcNumber
extern LocalPgBackendStatus *pgstat_get_local_beentry_by_index(int idx);
extern char *pgstat_clip_activity(const char *raw_activity);
+/*
+ * Fuctions to count memory
+ */
+extern void *malloc_and_count(Size size);
+extern void *realloc_and_count(void *ptr, Size new_size, Size old_size);
+extern void free_and_count(void *ptr, Size size);
+extern Datum pg_get_backend_memory_contexts_total_bytes(PG_FUNCTION_ARGS);
+extern Datum pg_get_backend_memory_allocation_stats(PG_FUNCTION_ARGS);
#endif /* BACKEND_STATUS_H */
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index c0d3cf0e14..f011f29b27 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -54,6 +54,10 @@ else
ALWAYS_SUBDIRS += ssl_passphrase_callback
endif
+ifneq (test_backend_memory,$(filter test_backend_memory,$(PG_TEST_SKIP)))
+SUBDIRS += test_backend_memory
+endif
+
# Test runs an LDAP server, so only run if ldap is in PG_TEST_EXTRA
ifeq ($(with_ldap),yes)
ifneq (,$(filter ldap,$(PG_TEST_EXTRA)))
diff --git a/src/test/modules/test_backend_memory/.gitignore b/src/test/modules/test_backend_memory/.gitignore
new file mode 100644
index 0000000000..5dcb3ff972
--- /dev/null
+++ b/src/test/modules/test_backend_memory/.gitignore
@@ -0,0 +1,4 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
diff --git a/src/test/modules/test_backend_memory/Makefile b/src/test/modules/test_backend_memory/Makefile
new file mode 100644
index 0000000000..c6afaedd0a
--- /dev/null
+++ b/src/test/modules/test_backend_memory/Makefile
@@ -0,0 +1,14 @@
+# src/test/modules/test_backend_memory/Makefile
+
+TAP_TESTS = 1
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_backend_memory
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_backend_memory/README b/src/test/modules/test_backend_memory/README
new file mode 100644
index 0000000000..95ffe2c438
--- /dev/null
+++ b/src/test/modules/test_backend_memory/README
@@ -0,0 +1 @@
+This directory contain tests for max_backend_memory GUC
diff --git a/src/test/modules/test_backend_memory/t/001_max_backend_memory.pl b/src/test/modules/test_backend_memory/t/001_max_backend_memory.pl
new file mode 100644
index 0000000000..c479f6ec4f
--- /dev/null
+++ b/src/test/modules/test_backend_memory/t/001_max_backend_memory.pl
@@ -0,0 +1,85 @@
+# Tests to check backend memory limit (max_backend_memory GUC)
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->start;
+
+# Initialize backend memory limit
+$node->safe_psql('postgres', qq{
+ ALTER SYSTEM SET max_backend_memory = 15;
+ SELECT pg_reload_conf();
+});
+
+# Check backend memory limit after SET.
+my $psql_stdout;
+$psql_stdout = $node->safe_psql('postgres',
+ "SELECT current_setting('max_backend_memory');");
+is($psql_stdout, '15MB', "max_backend_memory is SET correctly");
+
+# Create test table and function.
+$node->safe_psql('postgres', q{
+ CREATE TABLE test(t text);
+ INSERT INTO test VALUES (repeat('1234567890', 400000));
+
+ -- Recursive function that should cause overflow
+ CREATE FUNCTION test_func() RETURNS void LANGUAGE plpgsql AS $$
+ DECLARE
+ bt text;
+ BEGIN
+ SELECT t || 'x' FROM test INTO bt;
+ PERFORM test_func();
+ END;
+$$;
+});
+
+# Call function "test_func" several times for memory leak check.
+my $psql_stdout_first;
+for (my $i = 0; $i < 4; $i++)
+{
+ # Call function and check that it finishes with 'out of memory' error.
+ my ($ret, $stdout, $stderr) = $node->psql('postgres',
+ "SELECT test_func();");
+ is($ret, 3, 'recursive function call causes overflow');
+ like($stderr, qr/out of memory/, 'expected out of memory error');
+
+ $psql_stdout = $node->safe_psql('postgres',
+ "SELECT pg_get_backend_memory_contexts_total_bytes();");
+ if ($i eq 0)
+ {
+ # Store first value of backend_memory_contexts_total_bytes.
+ $psql_stdout_first = $psql_stdout;
+ is($psql_stdout_first > 0, 1,
+ "backend has allocated $psql_stdout_first (greater than 0) bytes");
+ next;
+ }
+ # Check other values of backend_memory_contexts_total_bytes.
+ # They should be the same as first value.
+ is($psql_stdout, $psql_stdout_first, "memory does not leak");
+}
+
+# Drop test table and function.
+$node->safe_psql('postgres', q{
+ DROP FUNCTION test_func();
+ DROP TABLE test;
+});
+
+# Deinitialize backend memory limit.
+$node->safe_psql('postgres', q{
+ ALTER SYSTEM RESET max_backend_memory;
+ SELECT pg_reload_conf();
+});
+
+# Check backend memory limit after RESET.
+$psql_stdout = $node->safe_psql('postgres',
+ "SELECT current_setting('max_backend_memory');");
+is($psql_stdout, '0', "max_backend_memory is RESET correctly");
+
+$node->stop;
+
+done_testing();
--
2.47.1
view thread (45+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Add the ability to limit the amount of memory that can be allocated to backends.
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox