From: torikoshia <torikoshia@oss.nttdata.com>
To: Fujii Masao <masao.fujii@oss.nttdata.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us
Cc: gkokolatos@protonmail.com
Cc: kasahara.tatsuhito@gmail.com
Cc: craig@2ndquadrant.com
Subject: Re: Get memory contexts of an arbitrary backend process
Date: Tue, 30 Mar 2021 22:06:58 +0900
Message-ID: <bbecd722d4f8e261b350186ac4bf68a7@oss.nttdata.com> (raw)
In-Reply-To: <168dbd34-c081-b832-e795-ffcd7c5be35c@oss.nttdata.com>
References: <20210326.131721.583838520933682740.horikyota.ntt@gmail.com>
<20210326.132836.985270106396322831.horikyota.ntt@gmail.com>
<e4cffb70-a672-fc7f-ef41-4c979c509832@oss.nttdata.com>
<20210326.140851.1169798659519899419.horikyota.ntt@gmail.com>
<fd78be66959cbf3957a329bb069ab737@oss.nttdata.com>
<168dbd34-c081-b832-e795-ffcd7c5be35c@oss.nttdata.com>
On 2021-03-30 02:28, Fujii Masao wrote:
Thanks for reviewing and kind suggestions!
>> It adds pg_log_backend_memory_contexts(pid) which logs memory contexts
>> of the specified backend process.
>>
>> The number of child contexts to be logged per parent is limited to 100
>> as with MemoryContextStats().
>>
>> As written in commit 7b5ef8f2d07, which limits the verbosity of
>> memory context statistics dumps, it supposes that practical cases
>> where the dump gets long will typically be huge numbers of
>> siblings under the same parent context; while the additional
>> debugging value from seeing details about individual siblings
>> beyond 100 will not be large.
>>
>> Thoughts?
>
> I'm OK with 100. We should comment why we chose 100 for that.
Added following comments.
+ /*
+ * When a backend process is consuming huge memory, logging all
its
+ * memory contexts might overrun available disk space. To
prevent
+ * this, we limit the number of child contexts per parent to
100.
+ *
+ * As with MemoryContextStats(), we suppose that practical cases
+ * where the dump gets long will typically be huge numbers of
+ * siblings under the same parent context; while the additional
+ * debugging value from seeing details about individual siblings
+ * beyond 100 will not be large.
+ */
+ MemoryContextStatsDetail(TopMemoryContext, 100, false);
>
> Here are some review comments.
>
> Isn't it better to move HandleProcSignalLogMemoryContext() and
> ProcessLogMemoryContextInterrupt() to mcxt.c from procsignal.c
> (like the functions for notify interrupt are defined in async.c)
> because they are the functions for memory contexts?
Agreed.
Also renamed HandleProcSignalLogMemoryContext to
HandleLogMemoryContextInterrupt.
> + * HandleProcSignalLogMemoryContext
> + *
> + * Handle receipt of an interrupt indicating log memory context.
> + * Signal handler portion of interrupt handling.
>
> IMO it's better to comment why we need to separate the function into
> two,
> i.e., HandleProcSignalLogMemoryContext() and
> ProcessLogMemoryContextInterrupt(),
> like the comment for other similar function explains. What about the
> followings?
Thanks! Changed them to the suggested one.
> -------------------------------
> HandleLogMemoryContextInterrupt
>
> Handle receipt of an interrupt indicating logging of memory contexts.
>
> All the actual work is deferred to ProcessLogMemoryContextInterrupt(),
> because we cannot safely emit a log message inside the signal handler.
> -------------------------------
> ProcessLogMemoryContextInterrupt
>
> Perform logging of memory contexts of this backend process.
>
> Any backend that participates in ProcSignal signaling must arrange to
> call this function if we see LogMemoryContextPending set. It is called
> from CHECK_FOR_INTERRUPTS(), which is enough because the target process
> for logging of memory contexts is a backend.
> -------------------------------
>
>
> + if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
> + HandleProcSignalLogMemoryContext();
> +
> if (CheckProcSignal(PROCSIG_BARRIER))
> HandleProcSignalBarrierInterrupt();
>
> The code for memory context logging interrupt came after barrier
> interrupt
> in other places, e.g., procsignal.h. Why is this order of code
> different?
Fixed.
> +/*
> + * pg_log_backend_memory_contexts
> + * Print memory context of the specified backend process.
>
> Isn't it better to move pg_log_backend_memory_contexts() to mcxtfuncs.c
> from mcxt.c because this is the SQL function memory contexts?
Agreed.
> IMO we should comment why we allow only superuser to call this
> function.
> What about the following?
Thanks!
Modified the patch according to the suggestions.
> -----------------
> Signal a backend process to log its memory contexts.
>
> Only superusers are allowed to signal to log the memory contexts
> because allowing any users to issue this request at an unbounded rate
> would cause lots of log messages and which can lead to denial of
> service.
> -----------------
>
> + PGPROC *proc = BackendPidGetProc(pid);
> +
> + /* Check whether the target process is PostgreSQL backend process. */
> + if (proc == NULL)
>
> What about adding more comments as follows?
>
> ---------------------------------
> + /*
> + * BackendPidGetProc returns NULL if the pid isn't valid; but by the
> time
> + * we reach kill(), a process for which we get a valid proc here
> might
> + * have terminated on its own. There's no way to acquire a lock on
> an
> + * arbitrary process to prevent that. But since this mechanism is
> usually
> + * used to debug a backend running and consuming lots of memory,
> + * that it might end on its own first and its memory contexts are not
> + * logged is not a problem.
> + */
> + if (proc == NULL)
> + {
> + /*
> + * This is just a warning so a loop-through-resultset will not abort
> + * if one backend logged its memory contexts during the run.
> + */
> + ereport(WARNING,
> ---------------------------------
>
> + ereport(ERROR,
> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> + errmsg("must be a superuser to log memory contexts")));
> + PG_RETURN_BOOL(false);
>
> This PG_RETURN_BOOL(false) is unnecessary because ereport() will emit
> an ERROR?
>
> +$node->psql('postgres', 'select
> pg_log_backend_memory_contexts(pg_backend_pid())');
> +
> +my $logfile = slurp_file($node->logfile);
> +like($logfile, qr/Grand total: \d+ bytes in \d+ blocks/,
> + 'found expected memory context print in the log file');
>
> Isn't there the case where the memory contexts have not been logged yet
> when slurp_file() is called? It's rare, though. If there is the risk
> for that,
> we should wait for the memory contexts to be actually logged?
>
> BTW, I agree that we should have the regression test that calls
> pg_log_backend_memory_contexts() and checks, e.g., that it doesn't
> cause
> segmentation fault. But I'm just wondering if it's a bit overkill to
> make perl
> scriptand start new node to test only this function...
OK, I removed the perl TAP test and added a regression test running
pg_log_backend_memory_contexts(pg_backend_pid()) and checking it returns
't'.
Regards.
Attachments:
[text/x-diff] v6-0001-add-memorycontext-elog-print.patch (26.2K, ../bbecd722d4f8e261b350186ac4bf68a7@oss.nttdata.com/2-v6-0001-add-memorycontext-elog-print.patch)
download | inline diff:
From 76a0be1a697939e2568ec39eb6365cffd79569bf Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Tue, 30 Mar 2021 01:58:18 +0900
Subject: [PATCH] After commit 3e98c0bafb28de, we can display the usage of
memory contexts using pg_backend_memory_contexts system view. However, its
target process is limited to the backend which is showing the view. This
patch introduces pg_log_backend_memory_contexts(pid) which logs memory
contexts of the specified backend process.
Currently the number of child contexts to be logged per parent is limited
to 100.
As with MemoryContextStats(), it supposes that practical cases where the
dump gets long will typically be huge numbers of siblings under the same
parent context; while the additional debugging value from seeing details
about individual siblings beyond 100 will not be large.
---
doc/src/sgml/func.sgml | 20 +++
src/backend/storage/ipc/procsignal.c | 4 +
src/backend/tcop/postgres.c | 3 +
src/backend/utils/adt/mcxtfuncs.c | 57 ++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/mmgr/aset.c | 8 +-
src/backend/utils/mmgr/generation.c | 8 +-
src/backend/utils/mmgr/mcxt.c | 163 +++++++++++++++----
src/backend/utils/mmgr/slab.c | 9 +-
src/include/catalog/pg_proc.dat | 6 +
src/include/miscadmin.h | 1 +
src/include/nodes/memnodes.h | 6 +-
src/include/storage/procsignal.h | 3 +
src/include/utils/memutils.h | 4 +-
src/test/regress/expected/misc_functions.out | 13 ++
src/test/regress/sql/misc_functions.sql | 9 +
16 files changed, 270 insertions(+), 45 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgmlindex fbf6062d0a..489ef5cd95 100644--- a/doc/src/sgml/func.sgml+++ b/doc/src/sgml/func.sgml@@ -24832,6 +24832,26 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>+ <entry role="func_table_entry"><para role="func_signature">+ <indexterm>+ <primary>pg_log_backend_memory_contexts</primary>+ </indexterm>+ <function>pg_log_backend_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> )+ <returnvalue>boolean</returnvalue>+ </para>+ <para>+ Log the memory contexts whose backend process has the specified+ process ID.+ Memory contexts will be logged based on the log configuration set.+ See <xref linkend="runtime-config-logging"/> for more information.+ The number of child contexts per parent is limited to 100.+ For more than 100 contexts, a summary will be shown.+ Only superusers can log the memory contexts even when the specified+ process is non-superuser backend.+ </para></entry>+ </row>+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.cindex c6a8d4611e..eac6895141 100644--- a/src/backend/storage/ipc/procsignal.c+++ b/src/backend/storage/ipc/procsignal.c@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -657,6 +658,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_BARRIER))
HandleProcSignalBarrierInterrupt();
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))+ HandleLogMemoryContextInterrupt();+
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_DATABASE))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_DATABASE);
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.cindex 2b1b68109f..afaf3b1cce 100644--- a/src/backend/tcop/postgres.c+++ b/src/backend/tcop/postgres.c@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
++ if (LogMemoryContextPending)+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.cindex c02fa47550..2dbdab406a 100644--- a/src/backend/utils/adt/mcxtfuncs.c+++ b/src/backend/utils/adt/mcxtfuncs.c@@ -18,6 +18,8 @@
#include "funcapi.h"
#include "miscadmin.h"
#include "mb/pg_wchar.h"
+#include "storage/proc.h"+#include "storage/procarray.h"
#include "utils/builtins.h"
/* ----------
@@ -61,7 +63,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -155,3 +157,56 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
return (Datum) 0;
}
++/*+ * pg_log_backend_memory_contexts+ * Signal a backend process to log its memory contexts.+ *+ * Only superusers are allowed to signal to log the memory contexts+ * because allowing any users to issue this request at an unbounded+ * rate would cause lots of log messages and which can lead to+ * denial of service.+ */+Datum+pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)+{+ int pid = PG_GETARG_INT32(0);+ PGPROC *proc = BackendPidGetProc(pid);++ /*+ * BackendPidGetProc returns NULL if the pid isn't valid; but by the+ * time we reach kill(), a process for which we get a valid proc here+ * might have terminated on its own. There's no way to acquire a+ * lock on an arbitrary process to prevent that. But since this+ * mechanism is usually used to debug a backend running and consuming+ * lots of memory, that it might end on its own first and its memory+ * contexts are not logged is not a problem.+ */+ if (proc == NULL)+ {+ /*+ * This is just a warning so a loop-through-resultset will not abort+ * if one backend logged its memory contexts during the run.+ */+ ereport(WARNING,+ (errmsg("PID %d is not a PostgreSQL server process", pid)));++ PG_RETURN_BOOL(false);+ }++ /* Only allow superusers to log memory contexts. */+ if (!superuser())+ ereport(ERROR,+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),+ errmsg("must be a superuser to log memory contexts")));++ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))+ PG_RETURN_BOOL(true);+ else+ {+ ereport(WARNING,+ (errmsg("could not send signal to process %d: %m", pid)));++ PG_RETURN_BOOL(false);+ }+}diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.cindex 73e0a672ae..6c27065f96 100644--- a/src/backend/utils/init/globals.c+++ b/src/backend/utils/init/globals.c@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.cindex ec6c130d0f..980c092558 100644--- a/src/backend/utils/mmgr/aset.c+++ b/src/backend/utils/mmgr/aset.c@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output is stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.cindex 2b90034764..b286308aa4 100644--- a/src/backend/utils/mmgr/generation.c+++ b/src/backend/utils/mmgr/generation.c@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.cindex 84472b9158..f1ed61e5ad 100644--- a/src/backend/utils/mmgr/mcxt.c+++ b/src/backend/utils/mmgr/mcxt.c@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"+#include "storage/procarray.h"+#include "storage/procsignal.h"+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -55,9 +59,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);+ const char *stats_string,+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -499,7 +505,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +514,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);- fprintf(stderr,- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",- grand_totals.totalspace, grand_totals.nblocks,- grand_totals.freespace, grand_totals.freechunks,- grand_totals.totalspace - grand_totals.freespace);+ if (is_dst_stderr)+ fprintf(stderr,+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",+ grand_totals.totalspace, grand_totals.nblocks,+ grand_totals.freespace, grand_totals.freechunks,+ grand_totals.totalspace - grand_totals.freespace);+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",+ grand_totals.totalspace, grand_totals.nblocks,+ grand_totals.freespace, grand_totals.freechunks,+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +548,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals,+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,7 +561,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
@@ -560,11 +576,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
if (ichild < max_children)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);+ totals,+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);+ &local_totals,+ is_dst_stderr);
}
/* Deal with excess children */
@@ -574,16 +592,31 @@ MemoryContextStatsInternal(MemoryContext context, int level,
{
int i;
- for (i = 0; i <= level; i++)- fprintf(stderr, " ");- fprintf(stderr,- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",- ichild - max_children,- local_totals.totalspace,- local_totals.nblocks,- local_totals.freespace,- local_totals.freechunks,- local_totals.totalspace - local_totals.freespace);+ if (is_dst_stderr)+ {+ for (i = 0; i <= level; i++)+ fprintf(stderr, " ");+ fprintf(stderr,+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",+ ichild - max_children,+ local_totals.totalspace,+ local_totals.nblocks,+ local_totals.freespace,+ local_totals.freechunks,+ local_totals.totalspace - local_totals.freespace);+ }+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",+ level,+ ichild - max_children,+ local_totals.totalspace,+ local_totals.nblocks,+ local_totals.freespace,+ local_totals.freechunks,+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,11 +638,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)+ const char *stats_string,+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
+ char truncated_ident[110];
int i;
/*
@@ -623,9 +658,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)- fprintf(stderr, " ");- fprintf(stderr, "%s: %s", name, stats_string);+ truncated_ident[0] = '\0';+
if (ident)
{
/*
@@ -637,24 +671,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");++ i = strlen(truncated_ident);+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';+
if (truncated)
- fprintf(stderr, "...");+ strcat(truncated_ident, "...");
}
- fputc('\n', stderr);++ if (is_dst_stderr)+ {+ for (i = 0; i < level; i++)+ fprintf(stderr, " ");+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);+ }+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("level: %d %s: %s%s",+ level, name, stats_string, truncated_ident)));
}
/*
@@ -946,6 +998,53 @@ MemoryContextAllocExtended(MemoryContext context, Size size, int flags)
return ret;
}
+/*+ * HandleLogMemoryContextInterrupt+ * Handle receipt of an interrupt indicating logging of memory+ * contexts.+ *+ * All the actual work is deferred to ProcessLogMemoryContextInterrupt(),+ * because we cannot safely emit a log message inside the signal handler.+ */+void+HandleLogMemoryContextInterrupt(void)+{+ InterruptPending = true;+ LogMemoryContextPending = true;+ /* latch will be set by procsignal_sigusr1_handler */+}++/*+ * ProcessLogMemoryContextInterrupt+ * Perform logging of memory contexts of this backend process.+ *+ * Any backend that participates in ProcSignal signaling must arrange+ * to call this function if we see LogMemoryContextPending set.+ * It is called from CHECK_FOR_INTERRUPTS(), which is enough because+ * the target process for logging of memory contexts is a backend.+ */+void+ProcessLogMemoryContextInterrupt(void)+{+ LogMemoryContextPending = false;++ ereport(LOG,+ (errmsg("Logging memory contexts of PID %d", MyProcPid)));++ /*+ * When a backend process is consuming huge memory, logging all its+ * memory contexts might overrun available disk space. To prevent+ * this, we limit the number of child contexts per parent to 100.+ *+ * As with MemoryContextStats(), we suppose that practical cases+ * where the dump gets long will typically be huge numbers of+ * siblings under the same parent context; while the additional+ * debugging value from seeing details about individual siblings+ * beyond 100 will not be large.+ */+ MemoryContextStatsDetail(TopMemoryContext, 100, false);+}+
void *
palloc(Size size)
{
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.cindex 9213be7c95..479b08bedd 100644--- a/src/backend/utils/mmgr/slab.c+++ b/src/backend/utils/mmgr/slab.c@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals,+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex bfb89e0575..8049587707 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7946,6 +7946,12 @@
proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# log memory contexts of the specified backend+{ oid => '4543', descr => 'log memory contexts of the specified backend',+ proname => 'pg_log_backend_memory_contexts',+ provolatile => 'v', prorettype => 'bool',+ proargtypes => 'int4', prosrc => 'pg_log_backend_memory_contexts' },+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.hindex 013850ac28..081822823c 100644--- a/src/include/miscadmin.h+++ b/src/include/miscadmin.h@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.hindex 9331ef80fd..87a8501c37 100644--- a/src/include/nodes/memnodes.h+++ b/src/include/nodes/memnodes.h@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);+ const char *stats_string,+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.hindex 4ae7dc33b8..55ea7104b3 100644--- a/src/include/storage/procsignal.h+++ b/src/include/storage/procsignal.h@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the+ memory contexts */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.hindex 36aae4e51c..1ebf5797cb 100644--- a/src/include/utils/memutils.h+++ b/src/include/utils/memutils.h@@ -84,7 +84,8 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
@@ -144,6 +145,7 @@ extern void MemoryContextCreate(MemoryContext node,
MemoryContext parent,
const char *name);
+extern void HandleLogMemoryContextInterrupt(void);
/*
* Memory-context-type-specific functions
diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.outindex d3acb98d04..e845042d38 100644--- a/src/test/regress/expected/misc_functions.out+++ b/src/test/regress/expected/misc_functions.out@@ -133,6 +133,19 @@ ERROR: function num_nulls() does not exist
LINE 1: SELECT num_nulls();
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
+--+-- pg_log_backend_memory_contexts()+--+-- Memory contexts are logged and they are not returned to the function.+-- Furthermore, their contents can vary depending on the timing. However,+-- we can at least verify that the code doesn't fail.+--+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());+ pg_log_backend_memory_contexts +--------------------------------+ t+(1 row)+
--
-- Test some built-in SRFs
--
diff --git a/src/test/regress/sql/misc_functions.sql b/src/test/regress/sql/misc_functions.sqlindex 094e8f8296..a398349afc 100644--- a/src/test/regress/sql/misc_functions.sql+++ b/src/test/regress/sql/misc_functions.sql@@ -30,6 +30,15 @@ SELECT num_nulls(VARIADIC '{}'::int[]);
SELECT num_nonnulls();
SELECT num_nulls();
+--+-- pg_log_backend_memory_contexts()+--+-- Memory contexts are logged and they are not returned to the function.+-- Furthermore, their contents can vary depending on the timing. However,+-- we can at least verify that the code doesn't fail.+--+SELECT * FROM pg_log_backend_memory_contexts(pg_backend_pid());+
--
-- Test some built-in SRFs
--
--
2.18.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: torikoshia@oss.nttdata.com, masao.fujii@oss.nttdata.com, horikyota.ntt@gmail.com, tgl@sss.pgh.pa.us, gkokolatos@protonmail.com, kasahara.tatsuhito@gmail.com, craig@2ndquadrant.com
Subject: Re: Get memory contexts of an arbitrary backend process
In-Reply-To: <bbecd722d4f8e261b350186ac4bf68a7@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox