From: torikoshia <torikoshia@oss.nttdata.com>
To: Kasahara Tatsuhito <kasahara.tatsuhito@gmail.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Fujii Masao <masao.fujii@oss.nttdata.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Creating a function for exposing memory usage of backend process
Date: Fri, 31 Jul 2020 17:24:44 +0900
Message-ID: <e1b81c88dc750ecd2ef03924af751567@oss.nttdata.com> (raw)
In-Reply-To: <CAP0=ZVJ2JPS41CffcG-E2CnBkfOjFRa4YoKB=A8vbWGkjEtA1w@mail.gmail.com>
References: <78268b5f-53ed-4645-1abb-23c1030cce0a@oss.nttdata.com>
<6c828f3a1d44dba81999fc7990b8f1c5@oss.nttdata.com>
<e068e42d-8eda-fbe5-e84c-7188e9386778@oss.nttdata.com>
<062bab2db5645812fae12e3a965de463@oss.nttdata.com>
<5cbed0cc-79fc-3b98-120a-20e07848d10a@oss.nttdata.com>
<805313b5f7dc2e6c19b209ff930a7a61@oss.nttdata.com>
<a9d68488-8343-5d70-4612-311d597f9962@oss.nttdata.com>
<cd0e961fd42e5708fdea70f7420bf214@oss.nttdata.com>
<9ee242b1-9e70-18f5-0033-6a086e71164d@oss.nttdata.com>
<3548926512ddfaedfb5eaf967bf2f196@oss.nttdata.com>
<20200708170348.rsqry4b4k5mrsu7y@alap3.anarazel.de>
<24ea05d7bbdb652feef50caee6fcfbf8@oss.nttdata.com>
<CAP0=ZVJ2JPS41CffcG-E2CnBkfOjFRa4YoKB=A8vbWGkjEtA1w@mail.gmail.com>
On 2020-07-30 15:13, Kasahara Tatsuhito wrote:
> Hi,
>
> On Fri, Jul 10, 2020 at 5:32 PM torikoshia <torikoshia@oss.nttdata.com>
> wrote:
>> - whether information for identifying parent-child relation is
>> necessary
>> or it's an overkill
> I think it's important to understand the parent-child relationship of
> the context.
> Personally, I often want to know the following two things ..
>
> - In which life cycle is the target context? (Remaining as long as the
> process is living? per query?)
> - Does the target context belong to the correct (parent) context?
>
>> - if this information is necessary, memory address is suitable or
>> other
>> means like assigning unique numbers are required
> IMO, If each context can be uniquely identified (or easily guessed) by
> "name" and "ident",
> then I don't think the address information is necessary.
> Instead, I like the way that directly shows the context name of the
> parent, as in the 0005 patch.
Thanks for your opinion!
I also feel it'll be sufficient to know not the exact memory context
of the parent but the name of the parent context.
And as Fujii-san told me in person, exposing memory address seems
not preferable considering there are security techniques like
address space layout randomization.
> On 2020-07-10 08:30:22 +0900, torikoshia wrote:
>> On 2020-07-08 22:12, Fujii Masao wrote:
>> Another comment about parent column is: dynahash can be parent?
>> If yes, its indent instead of name should be displayed in parent
>> column?
> I'm not sure yet, but considering the changes in the future, it seems
> better to do so.
Attached a patch which displays ident as parent when dynahash is a
parent.
I could not find the case when dynahash can be a parent so I tested it
using attached test purposed patch.
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:download all (2 files, .tar.gz)0007-Adding-a-function-exposing-memory-usage-of-local-backend.patch (text/x-diff, 12.4K) download
From 055af903a3dbf146d97dd3fb01a6a7d3d3bd2ae0 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Fri, 31 Jul 2020 16:20:29 +0900
Subject: [PATCH] Add a function exposing memory usage of local backend.
This patch implements a new SQL-callable function
pg_get_backend_memory_contexts which exposes memory usage of the
local backend.
It also adds a new view pg_backend_memory_contexts for exposing
local backend memory contexts.
---
doc/src/sgml/catalogs.sgml | 122 +++++++++++++++++++++++
src/backend/catalog/system_views.sql | 3 +
src/backend/utils/mmgr/mcxt.c | 140 +++++++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 9 ++
src/test/regress/expected/rules.out | 10 ++
5 files changed, 284 insertions(+)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgmlindex 26fda20d19..5bfc983a90 100644--- a/doc/src/sgml/catalogs.sgml+++ b/doc/src/sgml/catalogs.sgml@@ -9266,6 +9266,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>materialized views</entry>
</row>
+ <row>+ <entry><link linkend="view-pg-backend-memory-contexts"><structname>pg_backend_memory_contexts</structname></link></entry>+ <entry>backend memory contexts</entry>+ </row>+
<row>
<entry><link linkend="view-pg-policies"><structname>pg_policies</structname></link></entry>
<entry>policies</entry>
@@ -10544,6 +10549,123 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-backend-memory-contexts">+ <title><structname>pg_backend_memory_contexts</structname></title>++ <indexterm zone="view-pg-backend-memory-contexts">+ <primary>pg_backend_memory_contexts</primary>+ </indexterm>++ <para>+ The view <structname>pg_backend_memory_contexts</structname> displays all+ the local backend memory contexts.+ </para>+ <para>+ <structname>pg_backend_memory_contexts</structname> contains one row+ for each memory context.+ </para>++ <table>+ <title><structname>pg_backend_memory_contexts</structname> Columns</title>+ <tgroup cols="1">+ <thead>+ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ Column Type+ </para>+ <para>+ Description+ </para></entry>+ </row>+ </thead>++ <tbody>+ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>name</structfield> <type>text</type>+ </para>+ <para>+ Name of the memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>ident</structfield> <type>text</type>+ </para>+ <para>+ Identification information of the memory context. This field is truncated at 1024 bytes+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>parent</structfield> <type>text</type>+ </para>+ <para>+ Name of the parent of this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>level</structfield> <type>int4</type>+ </para>+ <para>+ Distance from TopMemoryContext in context tree+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>total_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Total bytes allocated for this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>total_nblocks</structfield> <type>int8</type>+ </para>+ <para>+ Total number of blocks allocated for this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>free_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Free space in bytes+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>free_chunks</structfield> <type>int8</type>+ </para>+ <para>+ Total number of free chunks+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>used_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Used space in bytes+ </para></entry>+ </row>+ </tbody>+ </tgroup>+ </table>++ </sect1>+
<sect1 id="view-pg-matviews">
<title><structname>pg_matviews</structname></title>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sqlindex 8625cbeab6..ba5a23ac25 100644--- a/src/backend/catalog/system_views.sql+++ b/src/backend/catalog/system_views.sql@@ -554,6 +554,9 @@ CREATE VIEW pg_shmem_allocations AS
REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+CREATE VIEW pg_backend_memory_contexts AS+ SELECT * FROM pg_get_backend_memory_contexts();+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.cindex abda22fa57..e4c69d90f6 100644--- a/src/backend/utils/mmgr/mcxt.c+++ b/src/backend/utils/mmgr/mcxt.c@@ -21,8 +21,10 @@
#include "postgres.h"
+#include "funcapi.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/builtins.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
@@ -67,6 +69,12 @@ static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
#define AssertNotInCriticalSection(context) \
Assert(CritSectionCount == 0 || (context)->allowInCritSection)
+/* ----------+ * The max bytes for showing identifiers of MemoryContext.+ * ----------+ */+#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024+
/*****************************************************************************
* EXPORTED ROUTINES *
*****************************************************************************/
@@ -1220,3 +1228,135 @@ pchomp(const char *in)
n--;
return pnstrdup(in, n);
}
++/*+ * PutMemoryContextsStatsTupleStore+ * One recursion level for pg_get_backend_memory_contexts.+ */+static void+PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,+ TupleDesc tupdesc, MemoryContext context,+ MemoryContext parent, int level)+{+#define PG_GET_BACKEND_MEMORY_CONTEXTS_COLS 9+ Datum values[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];+ bool nulls[PG_GET_BACKEND_MEMORY_CONTEXTS_COLS];+ MemoryContextCounters stat;+ MemoryContext child;+ const char *name = context->name;+ const char *ident = context->ident;++ if (context == NULL)+ return;++ /*+ * To be consistent with logging output, we label dynahash contexts+ * with just the hash table name as with MemoryContextStatsPrint().+ */+ if (ident && strcmp(name, "dynahash") == 0)+ {+ name = ident;+ ident = NULL;+ }++ /* Examine the context itself */+ memset(&stat, 0, sizeof(stat));+ (*context->methods->stats) (context, NULL, (void *) &level, &stat);++ memset(values, 0, sizeof(values));+ memset(nulls, 0, sizeof(nulls));++ values[0] = CStringGetTextDatum(name);++ if (ident)+ {+ int idlen = strlen(ident);+ char clipped_ident[MEMORY_CONTEXT_IDENT_DISPLAY_SIZE];++ /*+ * Some identifiers such as SQL query string can be very long,+ * truncate oversize identifiers.+ */+ if (idlen >= MEMORY_CONTEXT_IDENT_DISPLAY_SIZE)+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_DISPLAY_SIZE - 1);++ memcpy(clipped_ident, ident, idlen);+ clipped_ident[idlen] = '\0';+ values[1] = CStringGetTextDatum(clipped_ident);+ }+ else+ nulls[1] = true;++ if (parent == NULL)+ nulls[2] = true;+ else+ /*+ * We labeled dynahash contexts with just the hash table name.+ * To make it possible to identify its parent, we also display+ * parent's ident here.+ */+ if (parent->ident && strcmp(parent->name, "dynahash") == 0)+ values[2] = CStringGetTextDatum(parent->ident);+ else+ values[2] = CStringGetTextDatum(parent->name);++ values[3] = Int32GetDatum(level);+ values[4] = Int64GetDatum(stat.totalspace);+ values[5] = Int64GetDatum(stat.nblocks);+ values[6] = Int64GetDatum(stat.freespace);+ values[7] = Int64GetDatum(stat.freechunks);+ values[8] = Int64GetDatum(stat.totalspace - stat.freespace);+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);++ for (child = context->firstchild; child != NULL; child = child->nextchild)+ {+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,+ child, context, level + 1);+ }+}++/*+ * pg_get_backend_memory_contexts+ * SQL SRF showing backend memory context.+ */+Datum+pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)+{+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;+ TupleDesc tupdesc;+ Tuplestorestate *tupstore;+ MemoryContext per_query_ctx;+ MemoryContext oldcontext;++ /* check to see if caller supports us returning a tuplestore */+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("set-valued function called in context that cannot accept a set")));+ if (!(rsinfo->allowedModes & SFRM_Materialize))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("materialize mode required, but it is not allowed in this context")));++ /* Build a tuple descriptor for our result type */+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)+ elog(ERROR, "return type must be a row type");++ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;+ oldcontext = MemoryContextSwitchTo(per_query_ctx);++ tupstore = tuplestore_begin_heap(true, false, work_mem);+ rsinfo->returnMode = SFRM_Materialize;+ rsinfo->setResult = tupstore;+ rsinfo->setDesc = tupdesc;++ MemoryContextSwitchTo(oldcontext);++ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,+ TopMemoryContext, NULL, 0);++ /* clean up and return the tuplestore */+ tuplestore_donestoring(tupstore);++ return (Datum) 0;+}diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 082a11f270..27989971db 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7807,6 +7807,15 @@
proargnames => '{name,off,size,allocated_size}',
prosrc => 'pg_get_shmem_allocations' },
+# memory context of local backend+{ oid => '2282', descr => 'information about all memory contexts of local backend',+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',+ proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',+ proargmodes => '{o,o,o,o,o,o,o,o,o}',+ proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',+ prosrc => 'pg_get_backend_memory_contexts' },+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.outindex 601734a6f1..2a18dc423e 100644--- a/src/test/regress/expected/rules.out+++ b/src/test/regress/expected/rules.out@@ -1324,6 +1324,16 @@ pg_available_extensions| SELECT e.name,
e.comment
FROM (pg_available_extensions() e(name, default_version, comment)
LEFT JOIN pg_extension x ON ((e.name = x.extname)));
+pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,+ pg_get_backend_memory_contexts.ident,+ pg_get_backend_memory_contexts.parent,+ pg_get_backend_memory_contexts.level,+ pg_get_backend_memory_contexts.total_bytes,+ pg_get_backend_memory_contexts.total_nblocks,+ pg_get_backend_memory_contexts.free_bytes,+ pg_get_backend_memory_contexts.free_chunks,+ pg_get_backend_memory_contexts.used_bytes+ FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
TEST-make-child-of-dynahash.patch (text/x-diff, 993B) downloaddiff --git a/src/backend/utils/hash/dynahash.c b/src/backend/utils/hash/dynahash.cindex 5948b01abc..243e78bd1d 100644--- a/src/backend/utils/hash/dynahash.c+++ b/src/backend/utils/hash/dynahash.c@@ -337,6 +337,10 @@ hash_create(const char *tabname, long nelem, HASHCTL *info, int flags)
}
else
{
+ /* test */+ MemoryContext dyna_child;+ /* test end*/+
/* Create the hash table's private memory context */
if (flags & HASH_CONTEXT)
CurrentDynaHashCxt = info->hcxt;
@@ -345,6 +349,12 @@ hash_create(const char *tabname, long nelem, HASHCTL *info, int flags)
CurrentDynaHashCxt = AllocSetContextCreate(CurrentDynaHashCxt,
"dynahash",
ALLOCSET_DEFAULT_SIZES);
+ /* test */+ dyna_child = AllocSetContextCreate(CurrentDynaHashCxt,+ "dyna_child",+ ALLOCSET_DEFAULT_SIZES);+ dyna_child->ident = "dyna_child_ident";+ /* test end */
}
/* Initialize the hash header, plus a copy of the table name */
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: torikoshia@oss.nttdata.com, kasahara.tatsuhito@gmail.com, andres@anarazel.de, masao.fujii@oss.nttdata.com
Subject: Re: Creating a function for exposing memory usage of backend process
In-Reply-To: <e1b81c88dc750ecd2ef03924af751567@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox