From: torikoshia <torikoshia@oss.nttdata.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: masao.fujii@oss.nttdata.com
Cc: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us
Cc: gkokolatos@protonmail.com
Cc: kasahara.tatsuhito@gmail.com
Cc: craig@2ndquadrant.com
Subject: Re: Get memory contexts of an arbitrary backend process
Date: Thu, 25 Mar 2021 00:17:53 +0900
Message-ID: <0b0657d5febd0e46565a6bc9c62ba3f6@oss.nttdata.com> (raw)
In-Reply-To: <20210323.172433.1038800347387944240.horikyota.ntt@gmail.com>
References: <70ae4b79eb8b0dcf42161c80a00e3f22@oss.nttdata.com>
<49232304-1269-b2d4-c2ca-4b8683b29156@oss.nttdata.com>
<2322ae15a675ba13368491ced3ea5fb4@oss.nttdata.com>
<20210323.172433.1038800347387944240.horikyota.ntt@gmail.com>
On 2021-03-23 17:24, Kyotaro Horiguchi wrote:
Thanks for reviewing and suggestions!
> At Mon, 22 Mar 2021 15:09:58 +0900, torikoshia
> <torikoshia@oss.nttdata.com> wrote in
>> >> If MemoryContextStatsPrint(), i.e. showing 100 children at most is
>> >> enough, this hard limit may be acceptable.
>> > Can't this number be passed via shared memory?
>>
>> The attached patch uses static shared memory to pass the number.
>
> "pg_print_backend_memory_contexts"
>
> That name looks like as if it returns the result as text when used on
> command-line. We could have pg_get_backend_memory_context(bool
> dump_to_log (or where to dump), int limit). Or couldn't we name it
> differently even in the ase we add a separate function?
Redefined pg_get_backend_memory_contexts() as
pg_get_backend_memory_contexts(pid, int max_children).
When pid equals 0, pg_get_backend_memory_contexts() prints local memory
contexts as original pg_get_backend_memory_contexts() does.
In this case, 'max_children' is ignored.
When 'pid' does not equal 0 and it is the PID of the client backend,
memory contexts are logged through elog().
>
> +/*
> + * MaxChildrenPerContext
> + * Max number of children to print per one parent context.
> + */
> +int *MaxChildrenPerContext = NULL;
>
> Perhaps it'd be better to have a struct even if it consists only of
> one member. (Aligned) C-int values are atomic so we can omit the
> McxtPrintLock. (I don't think it's a problem even if it is modifed
> while reading^^:)
Fixed them.
> + if(max_children <= 0)
> + {
> + ereport(WARNING,
> + (errmsg("%d is invalid value",
> max_children),
> + errhint("second parameter is the number
> of context and it must
> be set to a value greater than or equal to 1")));
>
> It's annoying to choose a number large enough when I want to dump
> children unlimitedly. Couldn't we use 0 to specify "unlimited"?
Modified as you suggested.
> + (errmsg("%d is invalid value",
> max_children),
> + errhint("second parameter is the number
> of context and it must
> be set to a value greater than or equal to 1")));
>
> For the main message, (I think) we usually spell the "%d is invalid
> value" as "maximum number of children must be positive" or such. For
> the hint, we don't need a copy of the primary section of the
> documentation here.
Modified it to "The maximum number of children must be greater than 0".
>
> I think we should ERROR out for invalid parameters, at least for
> max_children. I'm not sure about pid since we might call it based on
> pg_stat_activity..
Changed to ERROR out when the 'max_children' is less than 0.
Regarding pid, I left it untouched considering the consistency with
other
signal sending functions such as pg_cancel_backend().
> + if(!SendProcSignal(pid, PROCSIG_PRINT_MEMORY_CONTEXT,
> InvalidBackendId))
>
> We know the backendid of the process here.
Added it.
>
> + if (is_dst_stderr)
> + {
> + for (i = 0; i <= level; i++)
> + fprintf(stderr, " ");
>
> The fprintf path is used nowhere in the patch at all. It can be used
> while attaching debugger but I'm not sure we need that code. The
> footprint of this patch is largely shrinked by removing it.
According to the past discussion[1], people wanted MemoryContextStats
as it was, so I think it's better that MemoryContextStats can be used
as before.
>
> + strcat(truncated_ident, delimiter);
>
> strcpy is sufficient here. And we don't need the delimiter to be a
> variable. (we can copy a string literal into truncate_ident, then
> count the length of truncate_ident, instead of the delimiter
> variable.)
True.
> + $current_logfiles = slurp_file($node->data_dir .
> '/current_logfiles');
> ...
> +my $lfname = $current_logfiles;
> +$lfname =~ s/^stderr //;
> +chomp $lfname;
>
> $node->logfile is the current log file name.
>
> + 'target PID is not PostgreSQL server process');
>
> Maybe "check if PID check is working" or such? And, we can do
> something like the following to exercise in a more practical way.
>
> select pg_print_backend...(pid,) from pg_stat_activity where
> backend_type = 'checkpointer';
It seems better.
>> As documented, the current implementation allows that when multiple
>> pg_print_backend_memory_contexts() called in succession or
>> simultaneously, max_children can be the one of another
>> pg_print_backend_memory_contexts().
>> I had tried to avoid this by adding some state information and using
>> before_shmem_exit() in case of process termination for cleaning up the
>> state information as in the patch I presented earlier, but since
>> kill()
>> returns success before the dumper called signal handler, it seemed
>> there were times when we couldn't clean up the state.
>> Since this happens only when multiple
>> pg_print_backend_memory_contexts()
>> are called and their specified number of children are different, and
>> the
>> effect is just the not intended number of children to print, it might
>> be
>> acceptable.
>
> I see it as a non-issue. Even though the behavior looks somewhat
> strange, that usage is stranger than the behavior.
Thanks for your comments!
>> Or it might be better to wait for some seconds if num_chilren on
>> shared
>> memory is not the initialized value(meaning some other process is
>> requesting to print memory contexts).
>>
>> >> Only superusers can call pg_print_backend_memory_contexts().
>> > + /* Only allow superusers to signal superuser-owned backends. */
>> > + if (superuser_arg(proc->roleId) && !superuser())
>> > The patch seems to allow even non-superuser to request to print the
>> > memory
>> > contexts if the target backend is owned by non-superuser. Is this
>> > intentional?
>> > I think that only superuser should be allowed to execute
>> > pg_print_backend_memory_contexts() whoever owns the target backend.
>> > Because that function can cause lots of log messages.
>>
>> Thanks, it's not intentional, modified it.
>
> By the way we can send a procsig to other than a client backend. And
> most of the postgres processes are using the standard signal handler
> and intializes the procsig facility. So some of such kind of processes
> can dump the memory context information as-is. Otherwise we can add
> CHECK_FOR_INTERRUPT to appropriate place to allow that. I'm not sure
> how it is useful for other kind of processes, but it might be useful
> for autovacuum workers.
Yeah, I also think it's possible to get memory contexts other than the
client backend.
But I'm not sure whether people want to do it...
[1] https://www.postgresql.org/message-id/906.1513707472%40sss.pgh.pa.us
regards.
Attachments:
[text/x-diff] v3-0001-add-memorycontext-elog-print.patch (30.4K, ../0b0657d5febd0e46565a6bc9c62ba3f6@oss.nttdata.com/2-v3-0001-add-memorycontext-elog-print.patch)
download | inline diff:diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgmlindex 1d3429fbd9..a4017a0760 100644--- a/doc/src/sgml/func.sgml+++ b/doc/src/sgml/func.sgml@@ -24821,6 +24821,37 @@ SELECT collation for ('foo' COLLATE "de_DE");
</para></entry>
</row>
+ <row>+ <entry role="func_table_entry"><para role="func_signature">+ <indexterm>+ <primary>pg_get_backend_memory_contexts</primary>+ </indexterm>+ <function>pg_get_backend_memory_contexts</function> (+ <parameter>pid</parameter> <type>integer</type>,+ <parameter>max_children</parameter> <type>integer</type> )+ <returnvalue>setof record</returnvalue>+ </para>+ <para>+ Get memory contexts whose backend process has the specified process ID.+ <parameter>max_children</parameter> limits the max number of children+ to print per one parent context. 0 means unlimited.+ When <parameter>pid</parameter> equals 0,+ <function>pg_get_backend_memory_contexts</function> displays all+ the memory contexts of the local process regardless of+ <parameter>max_children</parameter>. + When <parameter>pid</parameter> does not equal 0,+ memory contexts will be printed based on the log configuration set.+ See <xref linkend="runtime-config-logging"/> for more information.+ Only superusers can call this function even when the specified process+ is non-superuser backend.+ Note that when multiple+ <function>pg_get_backend_memory_contexts</function> called in+ succession or simultaneously, <parameter>max_children</parameter> can+ be the one of another+ <function>pg_get_backend_memory_contexts</function>.+ </para></entry>+ </row>+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sqlindex 0dca65dc7b..48a1a0e958 100644--- a/src/backend/catalog/system_views.sql+++ b/src/backend/catalog/system_views.sql@@ -555,10 +555,10 @@ REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
CREATE VIEW pg_backend_memory_contexts AS
- SELECT * FROM pg_get_backend_memory_contexts();+ SELECT * FROM pg_get_backend_memory_contexts(0, 0);
REVOKE ALL ON pg_backend_memory_contexts FROM PUBLIC;
-REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;+REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts FROM PUBLIC;
-- Statistics views
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.cindex 3e4ec53a97..ed5393324a 100644--- a/src/backend/storage/ipc/ipci.c+++ b/src/backend/storage/ipc/ipci.c@@ -46,6 +46,7 @@
#include "storage/sinvaladt.h"
#include "storage/spin.h"
#include "utils/snapmgr.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -269,6 +270,7 @@ CreateSharedMemoryAndSemaphores(void)
BTreeShmemInit();
SyncScanShmemInit();
AsyncShmemInit();
+ McxtLogShmemInit();
#ifdef EXEC_BACKEND
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.cindex c6a8d4611e..c61d5079e2 100644--- a/src/backend/storage/ipc/procsignal.c+++ b/src/backend/storage/ipc/procsignal.c@@ -30,6 +30,7 @@
#include "storage/shmem.h"
#include "storage/sinval.h"
#include "tcop/tcopprot.h"
+#include "utils/memutils.h"
/*
* The SIGUSR1 signal is multiplexed to support signaling multiple event
@@ -440,6 +441,20 @@ HandleProcSignalBarrierInterrupt(void)
/* latch will be set by procsignal_sigusr1_handler */
}
+/*+ * HandleProcSignalLogMemoryContext+ *+ * Handle receipt of an interrupt indicating logging memory context.+ * Signal handler portion of interrupt handling.+ */+static void+HandleProcSignalLogMemoryContext(void)+{+ InterruptPending = true;+ LogMemoryContextPending = true;+ /* latch will be set by procsignal_sigusr1_handler */+}+
/*
* Perform global barrier related interrupt checking.
*
@@ -580,6 +595,27 @@ ProcessProcSignalBarrier(void)
ConditionVariableBroadcast(&MyProcSignalSlot->pss_barrierCV);
}
+/*+ * ProcessLogMemoryContextInterrupt+ * The portion of logging memory context interrupt handling that runs+ * outside of the signal handler.+ */+void+ProcessLogMemoryContextInterrupt(void)+{+ int max_children;++ LogMemoryContextPending = false;++ max_children = mcxtLogData->maxChildrenPerContext;++ ereport(LOG,+ (errmsg("Logging memory contexts of PID %d. Max number of children per context is %d", MyProcPid, max_children)));++ MemoryContextStatsDetail(TopMemoryContext, max_children, false);+}++
/*
* If it turns out that we couldn't absorb one or more barrier types, either
* because the barrier-processing functions returned false or due to an error,
@@ -675,6 +711,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN))
RecoveryConflictInterrupt(PROCSIG_RECOVERY_CONFLICT_BUFFERPIN);
+ if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))+ HandleProcSignalLogMemoryContext();+
SetLatch(MyLatch);
errno = save_errno;
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.cindex 2b1b68109f..afaf3b1cce 100644--- a/src/backend/tcop/postgres.c+++ b/src/backend/tcop/postgres.c@@ -3295,6 +3295,9 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
++ if (LogMemoryContextPending)+ ProcessLogMemoryContextInterrupt();
}
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.cindex c02fa47550..de2040640c 100644--- a/src/backend/utils/adt/mcxtfuncs.c+++ b/src/backend/utils/adt/mcxtfuncs.c@@ -19,6 +19,7 @@
#include "miscadmin.h"
#include "mb/pg_wchar.h"
#include "utils/builtins.h"
+#include "utils/mcxtfuncs.h"
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -61,7 +62,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
/* Examine the context itself */
memset(&stat, 0, sizeof(stat));
- (*context->methods->stats) (context, NULL, (void *) &level, &stat);+ (*context->methods->stats) (context, NULL, (void *) &level, &stat, true);
memset(values, 0, sizeof(values));
memset(nulls, 0, sizeof(nulls));
@@ -117,6 +118,9 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
Datum
pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
{
+ int pid = PG_GETARG_INT32(0);+ int max_children = PG_GETARG_INT32(1);+
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -147,8 +151,11 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
MemoryContextSwitchTo(oldcontext);
- PutMemoryContextsStatsTupleStore(tupstore, tupdesc,- TopMemoryContext, NULL, 0);+ if( pid == 0)+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,+ TopMemoryContext, NULL, 0);+ else+ pg_log_backend_memory_contexts(pid, max_children);
/* clean up and return the tuplestore */
tuplestore_donestoring(tupstore);
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.cindex 73e0a672ae..6c27065f96 100644--- a/src/backend/utils/init/globals.c+++ b/src/backend/utils/init/globals.c@@ -34,6 +34,7 @@ volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
+volatile sig_atomic_t LogMemoryContextPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
volatile uint32 CritSectionCount = 0;
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.cindex ec6c130d0f..8e7fe09a0f 100644--- a/src/backend/utils/mmgr/aset.c+++ b/src/backend/utils/mmgr/aset.c@@ -272,7 +272,8 @@ static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
static bool AllocSetIsEmpty(MemoryContext context);
static void AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void AllocSetCheck(MemoryContext context);
@@ -1336,11 +1337,12 @@ AllocSetIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals, bool is_dst_stderr)
{
AllocSet set = (AllocSet) context;
Size nblocks = 0;
@@ -1379,7 +1381,7 @@ AllocSetStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.cindex 2b90034764..b286308aa4 100644--- a/src/backend/utils/mmgr/generation.c+++ b/src/backend/utils/mmgr/generation.c@@ -155,7 +155,8 @@ static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
static bool GenerationIsEmpty(MemoryContext context);
static void GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void GenerationCheck(MemoryContext context);
@@ -665,6 +666,7 @@ GenerationIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
@@ -672,7 +674,7 @@ GenerationIsEmpty(MemoryContext context)
static void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals, bool is_dst_stderr)
{
GenerationContext *set = (GenerationContext *) context;
Size nblocks = 0;
@@ -704,7 +706,7 @@ GenerationStats(MemoryContext context,
"%zu total in %zd blocks (%zd chunks); %zu free (%zd chunks); %zu used",
totalspace, nblocks, nchunks, freespace,
nfreechunks, totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.cindex 84472b9158..f02331c36c 100644--- a/src/backend/utils/mmgr/mcxt.c+++ b/src/backend/utils/mmgr/mcxt.c@@ -23,6 +23,10 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "storage/proc.h"+#include "storage/procarray.h"+#include "storage/procsignal.h"+#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -37,6 +41,13 @@
*/
MemoryContext CurrentMemoryContext = NULL;
+/*+ * McxtLogData+ * Data for communication between memory context logging+ * dumper and requestor.+ */+McxtLogData *mcxtLogData = NULL;+
/*
* Standard top-level contexts. For a description of the purpose of each
* of these contexts, refer to src/backend/utils/mmgr/README
@@ -55,9 +66,11 @@ MemoryContext PortalContext = NULL;
static void MemoryContextCallResetCallbacks(MemoryContext context);
static void MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string);+ const char *stats_string,+ bool is_dst_stderr);
/*
* You should not do memory allocations within a critical section, because
@@ -487,6 +500,84 @@ MemoryContextMemAllocated(MemoryContext context, bool recurse)
return total;
}
+/*+ * McxtLogShmemInit+ * Initialize shared memory area+ */+void+McxtLogShmemInit(void)+{+ bool found;++ mcxtLogData = (McxtLogData *)+ ShmemInitStruct("Memory Context Logging Data",+ sizeof(int),+ &found);+ if (!found)+ {+ mcxtLogData->maxChildrenPerContext = 0;+ }+}++/*+ * pg_log_backend_memory_contexts+ * Log memory contexts of the specified backend process.+ */+Datum+pg_log_backend_memory_contexts(int pid, int max_children)+{+ PGPROC *proc = BackendPidGetProc(pid);++ if(max_children < 0)+ {+ ereport(ERROR,+ (errmsg("The maximum number of children must be greater than or equal to 0")));++ PG_RETURN_BOOL(false);+ }++ /*+ * Check whether the target process is PostgreSQL backend process.+ */+ if (proc == NULL)+ {+ /*+ * According to the thread below, it might be better to+ * change this message.+ *+ * https://www.postgresql.org/message-id/CALj2ACW7Rr-R7mBcBQiXWPp%3DJV5chajjTdudLiF5YcpW-BmHhg%40mail.gmail.com+ *+ * Until the conclusion of the discussion, output the same+ * message as pg_cancel_backend().+ */+ ereport(WARNING,+ (errmsg("PID %d is not a PostgreSQL server process", pid)));++ PG_RETURN_BOOL(false);+ }++ /* Only allow superusers to print memory contexts. */+ if (!superuser())+ {+ ereport(WARNING,+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),+ errmsg("must be a superuser to log memory contexts")));+ PG_RETURN_BOOL(false);+ }++ mcxtLogData->maxChildrenPerContext = max_children;++ if(!SendProcSignal(pid, PROCSIG_LOG_MEMORY_CONTEXT, proc->backendId))+ PG_RETURN_BOOL(true);+ else+ {+ ereport(WARNING,+ (errmsg("failed to send signal: %m")));++ PG_RETURN_BOOL(false);+ }+}+
/*
* MemoryContextStats
* Print statistics about the named context and all its descendants.
@@ -499,7 +590,7 @@ void
MemoryContextStats(MemoryContext context)
{
/* A hard-wired limit on the number of children is usually good enough */
- MemoryContextStatsDetail(context, 100);+ MemoryContextStatsDetail(context, 100, true);
}
/*
@@ -508,19 +599,28 @@ MemoryContextStats(MemoryContext context)
* Entry point for use if you want to vary the number of child contexts shown.
*/
void
-MemoryContextStatsDetail(MemoryContext context, int max_children)+MemoryContextStatsDetail(MemoryContext context, int max_children, bool is_dst_stderr)
{
MemoryContextCounters grand_totals;
memset(&grand_totals, 0, sizeof(grand_totals));
- MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals);+ MemoryContextStatsInternal(context, 0, true, max_children, &grand_totals, is_dst_stderr);- fprintf(stderr,- "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",- grand_totals.totalspace, grand_totals.nblocks,- grand_totals.freespace, grand_totals.freechunks,- grand_totals.totalspace - grand_totals.freespace);+ if (is_dst_stderr)+ fprintf(stderr,+ "Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used\n",+ grand_totals.totalspace, grand_totals.nblocks,+ grand_totals.freespace, grand_totals.freechunks,+ grand_totals.totalspace - grand_totals.freespace);+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("Grand total: %zu bytes in %zd blocks; %zu free (%zd chunks); %zu used",+ grand_totals.totalspace, grand_totals.nblocks,+ grand_totals.freespace, grand_totals.freechunks,+ grand_totals.totalspace - grand_totals.freespace)));
}
/*
@@ -533,7 +633,8 @@ MemoryContextStatsDetail(MemoryContext context, int max_children)
static void
MemoryContextStatsInternal(MemoryContext context, int level,
bool print, int max_children,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals,+ bool is_dst_stderr)
{
MemoryContextCounters local_totals;
MemoryContext child;
@@ -545,11 +646,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
context->methods->stats(context,
print ? MemoryContextStatsPrint : NULL,
(void *) &level,
- totals);+ totals, is_dst_stderr);
/*
* Examine children. If there are more than max_children of them, we do
* not print the rest explicitly, but just summarize them.
+ *+ * Note that specifying 0 to max_children means umlimited.
*/
memset(&local_totals, 0, sizeof(local_totals));
@@ -557,33 +660,50 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child != NULL;
child = child->nextchild, ichild++)
{
- if (ichild < max_children)+ if (ichild < max_children || max_children == 0)
MemoryContextStatsInternal(child, level + 1,
print, max_children,
- totals);+ totals,+ is_dst_stderr);
else
MemoryContextStatsInternal(child, level + 1,
false, max_children,
- &local_totals);+ &local_totals,+ is_dst_stderr);
}
- /* Deal with excess children */- if (ichild > max_children)+ /*Deal with excess children */+ if (ichild > max_children && max_children != 0)
{
if (print)
{
int i;
- for (i = 0; i <= level; i++)- fprintf(stderr, " ");- fprintf(stderr,- "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",- ichild - max_children,- local_totals.totalspace,- local_totals.nblocks,- local_totals.freespace,- local_totals.freechunks,- local_totals.totalspace - local_totals.freespace);+ if (is_dst_stderr)+ {+ for (i = 0; i <= level; i++)+ fprintf(stderr, " ");+ fprintf(stderr,+ "%d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used\n",+ ichild - max_children,+ local_totals.totalspace,+ local_totals.nblocks,+ local_totals.freespace,+ local_totals.freechunks,+ local_totals.totalspace - local_totals.freespace);+ }+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("level: %d %d more child contexts containing %zu total in %zd blocks; %zu free (%zd chunks); %zu used",+ level,+ ichild - max_children,+ local_totals.totalspace,+ local_totals.nblocks,+ local_totals.freespace,+ local_totals.freechunks,+ local_totals.totalspace - local_totals.freespace)));
}
if (totals)
@@ -605,12 +725,14 @@ MemoryContextStatsInternal(MemoryContext context, int level,
*/
static void
MemoryContextStatsPrint(MemoryContext context, void *passthru,
- const char *stats_string)+ const char *stats_string,+ bool is_dst_stderr)
{
int level = *(int *) passthru;
const char *name = context->name;
const char *ident = context->ident;
int i;
+ char truncated_ident[110];
/*
* It seems preferable to label dynahash contexts with just the hash table
@@ -623,9 +745,8 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
ident = NULL;
}
- for (i = 0; i < level; i++)- fprintf(stderr, " ");- fprintf(stderr, "%s: %s", name, stats_string);+ truncated_ident[0] = '\0';+
if (ident)
{
/*
@@ -637,24 +758,42 @@ MemoryContextStatsPrint(MemoryContext context, void *passthru,
int idlen = strlen(ident);
bool truncated = false;
+ strcpy(truncated_ident, ": ");+
if (idlen > 100)
{
idlen = pg_mbcliplen(ident, idlen, 100);
truncated = true;
}
- fprintf(stderr, ": ");++ i = strlen(truncated_ident);+
while (idlen-- > 0)
{
unsigned char c = *ident++;
if (c < ' ')
c = ' ';
- fputc(c, stderr);+ truncated_ident[i++] = c;
}
+ truncated_ident[i] = '\0';+
if (truncated)
- fprintf(stderr, "...");+ strcat(truncated_ident, "...");+ }++ if (is_dst_stderr)+ {+ for (i = 0; i < level; i++)+ fprintf(stderr, " ");+ fprintf(stderr, "%s: %s%s\n", name, stats_string, truncated_ident);
}
- fputc('\n', stderr);+ else+ ereport(LOG_SERVER_ONLY,+ (errhidestmt(true),+ errhidecontext(true),+ errmsg_internal("level: %d %s: %s%s",+ level, name, stats_string, truncated_ident)));
}
/*
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.cindex 9213be7c95..479b08bedd 100644--- a/src/backend/utils/mmgr/slab.c+++ b/src/backend/utils/mmgr/slab.c@@ -135,7 +135,8 @@ static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
static bool SlabIsEmpty(MemoryContext context);
static void SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
static void SlabCheck(MemoryContext context);
#endif
@@ -632,11 +633,13 @@ SlabIsEmpty(MemoryContext context)
* printfunc: if not NULL, pass a human-readable stats string to this.
* passthru: pass this pointer through to printfunc.
* totals: if not NULL, add stats about this context into *totals.
+ * is_dst_stderr: is the output should be stderr or elog.
*/
static void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals)+ MemoryContextCounters *totals,+ bool is_dst_stderr)
{
SlabContext *slab = castNode(SlabContext, context);
Size nblocks = 0;
@@ -671,7 +674,7 @@ SlabStats(MemoryContext context,
"%zu total in %zd blocks; %zu free (%zd chunks); %zu used",
totalspace, nblocks, freespace, freechunks,
totalspace - freespace);
- printfunc(context, passthru, stats_string);+ printfunc(context, passthru, stats_string, is_dst_stderr);
}
if (totals)
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 464fa8d614..fa20889105 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7921,12 +7921,14 @@
descr => 'information about all memory contexts of local backend',
proname => 'pg_get_backend_memory_contexts', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 'r',
- prorettype => 'record', proargtypes => '',- proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',- proargmodes => '{o,o,o,o,o,o,o,o,o}',- proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',+ prorettype => 'record', proargtypes => 'int4 int4',+ proallargtypes => '{int4,int4,text,text,text,int4,int8,int8,int8,int8,int8}',+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o}',+ proargnames => '{pid, max_children, name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',
prosrc => 'pg_get_backend_memory_contexts' },
+# print memory context of specified backend+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.hindex 013850ac28..081822823c 100644--- a/src/include/miscadmin.h+++ b/src/include/miscadmin.h@@ -84,6 +84,7 @@ extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
+extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.hindex 9331ef80fd..87a8501c37 100644--- a/src/include/nodes/memnodes.h+++ b/src/include/nodes/memnodes.h@@ -52,7 +52,8 @@ typedef struct MemoryContextCounters
*/
typedef void (*MemoryStatsPrintFunc) (MemoryContext context, void *passthru,
- const char *stats_string);+ const char *stats_string,+ bool is_dst_stderr);
typedef struct MemoryContextMethods
{
@@ -66,7 +67,8 @@ typedef struct MemoryContextMethods
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
- MemoryContextCounters *totals);+ MemoryContextCounters *totals,+ bool is_dst_stderr);
#ifdef MEMORY_CONTEXT_CHECKING
void (*check) (MemoryContext context);
#endif
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.hindex 4ae7dc33b8..1144536031 100644--- a/src/include/storage/procsignal.h+++ b/src/include/storage/procsignal.h@@ -34,6 +34,8 @@ typedef enum
PROCSIG_PARALLEL_MESSAGE, /* message from cooperating parallel backend */
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
+ PROCSIG_LOG_MEMORY_CONTEXT, /* ask specified backend to log the+ memory context */
/* Recovery conflict reasons */
PROCSIG_RECOVERY_CONFLICT_DATABASE,
@@ -69,6 +71,7 @@ extern int SendProcSignal(pid_t pid, ProcSignalReason reason,
extern uint64 EmitProcSignalBarrier(ProcSignalBarrierType type);
extern void WaitForProcSignalBarrier(uint64 generation);
extern void ProcessProcSignalBarrier(void);
+extern void ProcessLogMemoryContextInterrupt(void);
extern void procsignal_sigusr1_handler(SIGNAL_ARGS);
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.hindex 36aae4e51c..9596aa7f14 100644--- a/src/include/utils/memutils.h+++ b/src/include/utils/memutils.h@@ -46,6 +46,12 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+typedef struct McxtLogData+{+ /* Max number of children to print per one parent context */+ int maxChildrenPerContext;+} McxtLogData;+
/*
* Standard top-level memory contexts.
*
@@ -63,6 +69,8 @@ extern PGDLLIMPORT MemoryContext CurTransactionContext;
/* This is a transient link to the active portal's memory context: */
extern PGDLLIMPORT MemoryContext PortalContext;
+extern McxtLogData *mcxtLogData;+
/* Backwards compatibility macro */
#define MemoryContextResetAndDeleteChildren(ctx) MemoryContextReset(ctx)
@@ -84,9 +92,12 @@ extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
extern Size MemoryContextMemAllocated(MemoryContext context, bool recurse);
extern void MemoryContextStats(MemoryContext context);
-extern void MemoryContextStatsDetail(MemoryContext context, int max_children);+extern void MemoryContextStatsDetail(MemoryContext context, int max_children,+ bool is_dst_stderr);
extern void MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow);
+extern void McxtLogShmemInit(void);+extern Datum pg_log_backend_memory_contexts(int pid, int max_children);
#ifdef MEMORY_CONTEXT_CHECKING
extern void MemoryContextCheck(MemoryContext context);
diff --git a/src/test/modules/test_misc/t/002_print_memory_context_validation.pl b/src/test/modules/test_misc/t/002_print_memory_context_validation.plnew file mode 100644index 0000000000..056ced1da2--- /dev/null+++ b/src/test/modules/test_misc/t/002_print_memory_context_validation.pl@@ -0,0 +1,90 @@+use strict;+use warnings;+use PostgresNode;+use TestLib;+use Test::More tests => 4;++# Set up node with logging collector+my $node = get_new_node('primary');+$node->init();+$node->append_conf(+ 'postgresql.conf', qq{+ logging_collector = on+ lc_messages = 'C'+ });++$node->start();++# Verify that log output gets to the file+$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), 100)');++# might need to retry if logging collector process is slow...+my $max_attempts = 180 * 10;++my $current_logfiles;+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)+{+ eval {+ $current_logfiles = slurp_file($node->data_dir . '/current_logfiles');+ };+ last unless $@;+ usleep(100_000);+}+die $@ if $@;++note "current_logfiles = $current_logfiles";++like(+ $current_logfiles,+ qr|^stderr log/postgresql-.*log$|,+ 'current_logfiles is sane');++my $lfname = $current_logfiles;+$lfname =~ s/^stderr //;+chomp $lfname;++my $logfile;+my $print_count;++# Verify that the backtraces of the processes are logged into logfile.+for (my $attempts = 0; $attempts < $max_attempts; $attempts++)+{+ $logfile = $node->data_dir . '/' . $lfname;+ chomp $logfile;+ print "file is $logfile";+ open my $fh, '<', $logfile+ or die "Could not open '$logfile' $!";+ while (my $line = <$fh>)+ {+ chomp $line;+ if ($line =~ m/Grand total: \d+ bytes in \d+ blocks/)+ {+ $print_count++;+ }+ }+ last if $print_count == 1;+ usleep(100_000);+}++is($print_count, 1, 'found expected memory context print in the log file');++my $output;++$node->psql('postgres', 'select pg_get_backend_memory_contexts(pg_backend_pid(), -1)',+ stderr => \$output);++like(+ $output,+ qr/The maximum number of children must be greater than or equal to 0/,+ 'check if max_children check is working');++$node->psql('postgres',+ 'select pg_get_backend_memory_contexts(pid, 100) from pg_stat_activity where backend_type = \'checkpointer\'',+ stderr => \$output);++like(+ $output,+ qr/PID \d+ is not a PostgreSQL server process/,+ 'check if PID check is working');++$node->stop('fast');diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.outindex 9b12cc122a..5841e5cc12 100644--- a/src/test/regress/expected/rules.out+++ b/src/test/regress/expected/rules.out@@ -1333,7 +1333,7 @@ pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,
pg_get_backend_memory_contexts.free_bytes,
pg_get_backend_memory_contexts.free_chunks,
pg_get_backend_memory_contexts.used_bytes
- FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);+ FROM pg_get_backend_memory_contexts(0, 0) pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: torikoshia@oss.nttdata.com, horikyota.ntt@gmail.com, masao.fujii@oss.nttdata.com, tgl@sss.pgh.pa.us, gkokolatos@protonmail.com, kasahara.tatsuhito@gmail.com, craig@2ndquadrant.com
Subject: Re: Get memory contexts of an arbitrary backend process
In-Reply-To: <0b0657d5febd0e46565a6bc9c62ba3f6@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox