From: torikoshia <torikoshia@oss.nttdata.com>
To: Fujii Masao <masao.fujii@oss.nttdata.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Creating a function for exposing memory usage of backend process
Date: Mon, 06 Jul 2020 12:12:18 +0900
Message-ID: <cd0e961fd42e5708fdea70f7420bf214@oss.nttdata.com> (raw)
In-Reply-To: <a9d68488-8343-5d70-4612-311d597f9962@oss.nttdata.com>
References: <72a656e0f71d0860161e0b3f67e4d771@oss.nttdata.com>
<398eb7cd-5f75-be1c-3897-36865b7bfabf@oss.nttdata.com>
<CA+TgmobkpFV0UB67kzXuD36--OFHwz1bs=L_6PZbD4nxKqUQMw@mail.gmail.com>
<3193c7d590f3212e543b63df977da758@oss.nttdata.com>
<78268b5f-53ed-4645-1abb-23c1030cce0a@oss.nttdata.com>
<6c828f3a1d44dba81999fc7990b8f1c5@oss.nttdata.com>
<e068e42d-8eda-fbe5-e84c-7188e9386778@oss.nttdata.com>
<062bab2db5645812fae12e3a965de463@oss.nttdata.com>
<5cbed0cc-79fc-3b98-120a-20e07848d10a@oss.nttdata.com>
<805313b5f7dc2e6c19b209ff930a7a61@oss.nttdata.com>
<a9d68488-8343-5d70-4612-311d597f9962@oss.nttdata.com>
On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao <masao.fujii@oss.nttdata.com>
wrote:
Thanks for your review!
> I like more specific name like pg_backend_memory_contexts.
Agreed.
When I was trying to add this function as statistics function,
I thought that naming pg_stat_getbackend_memory_context()
might make people regarded it as a "per-backend statistics
function", whose parameter is backend ID number.
So I removed "backend", but now there is no necessity to do
so.
> But I'd like to hear more opinions about the name from others.
I changed the name to pg_backend_memory_contexts for the time
being.
>> - function name: pg_get_memory_contexts()
>> - source file: mainly src/backend/utils/mmgr/mcxt.c
>> + Identification information of the memory context. This field
>> is truncated if the identification field is longer than 1024
>> characters
>
> "characters" should be "bytes"?
Fixed, but I used "characters" while referring to the
descriptions on the manual of pg_stat_activity.query
below.
| By default the query text is truncated at 1024 characters;
It has nothing to do with this thread, but considering
multibyte characters, it also may be better to change it
to "bytes".
Regarding the other comments, I revised the patch as you pointed.
Regards,
--
Atsushi Torikoshi
NTT DATA CORPORATION
Attachments:0004-Adding-a-function-exposing-memory-usage-of-local-backend.patch (text/x-diff, 12.5K) download
From 055af903a3dbf146d97dd3fb01a6a7d3d3bd2ae0 Mon Sep 17 00:00:00 2001
From: Atsushi Torikoshi <torikoshia@oss.nttdata.com>
Date: Mon, 6 Jul 2020 10:50:49 +0900
Subject: [PATCH] Add a function exposing memory usage of local backend.
Previously, the only way to examine the usage of backend processes
was attaching a debugger and call MemoryContextStats() and it was
not so convenient in some cases.
This patch implements a new SQL-callable function
pg_get_backend_memory_contexts which exposes memory usage of the
local backend.
It also adds a new view pg_backend_memory_contexts for exposing
local backend memory contexts.
---
doc/src/sgml/catalogs.sgml | 126 +++++++++++++++++++++++++
src/backend/catalog/system_views.sql | 3 +
src/backend/utils/mmgr/mcxt.c | 132 ++++++++++++++++++++++++++-
src/include/catalog/pg_proc.dat | 9 ++
src/test/regress/expected/rules.out | 10 ++
5 files changed, 279 insertions(+), 1 deletion(-)
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgmlindex 49a881b262..abc097179f 100644--- a/doc/src/sgml/catalogs.sgml+++ b/doc/src/sgml/catalogs.sgml@@ -9248,6 +9248,11 @@ SCRAM-SHA-256$<replaceable><iteration count></replaceable>:<replaceable>&l
<entry>materialized views</entry>
</row>
+ <row>+ <entry><link linkend="view-pg-backend-memory-contexts"><structname>pg_backend_memory_contexts</structname></link></entry>+ <entry>backend memory contexts</entry>+ </row>+
<row>
<entry><link linkend="view-pg-policies"><structname>pg_policies</structname></link></entry>
<entry>policies</entry>
@@ -10526,6 +10531,127 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</sect1>
+ <sect1 id="view-pg-backend-memory-contexts">+ <title><structname>pg_backend_memory_contexts</structname></title>++ <indexterm zone="view-pg-backend-memory-contexts">+ <primary>pg_backend_memory_contexts</primary>+ </indexterm>++ <indexterm zone="view-pg-backend-memory-contexts">+ <primary>backend memory contexts</primary>+ </indexterm>++ <para>+ The view <structname>pg_backend_memory_contexts</structname> displays all+ the local backend memory contexts.+ </para>+ <para>+ <structname>pg_backend_memory_contexts</structname> contains one row+ for each memory context.+ </para>++ <table>+ <title><structname>pg_backend_memory_contexts</structname> Columns</title>+ <tgroup cols="1">+ <thead>+ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ Column Type+ </para>+ <para>+ Description+ </para></entry>+ </row>+ </thead>++ <tbody>+ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>name</structfield> <type>text</type>+ </para>+ <para>+ Name of the memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>ident</structfield> <type>text</type>+ </para>+ <para>+ Identification information of the memory context. This field is truncated at 1024 bytes+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>parent</structfield> <type>text</type>+ </para>+ <para>+ Name of the parent of this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>level</structfield> <type>int4</type>+ </para>+ <para>+ Distance from TopMemoryContext in context tree+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>total_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Total bytes allocated for this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>total_nblocks</structfield> <type>int8</type>+ </para>+ <para>+ Total number of blocks allocated for this memory context+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>free_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Free space in bytes+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>free_chunks</structfield> <type>int8</type>+ </para>+ <para>+ Total number of free chunks+ </para></entry>+ </row>++ <row>+ <entry role="catalog_table_entry"><para role="column_definition">+ <structfield>used_bytes</structfield> <type>int8</type>+ </para>+ <para>+ Used space in bytes+ </para></entry>+ </row>+ </tbody>+ </tgroup>+ </table>++ </sect1>+
<sect1 id="view-pg-matviews">
<title><structname>pg_matviews</structname></title>
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sqlindex 5314e9348f..403954039b 100644--- a/src/backend/catalog/system_views.sql+++ b/src/backend/catalog/system_views.sql@@ -554,6 +554,9 @@ CREATE VIEW pg_shmem_allocations AS
REVOKE ALL ON pg_shmem_allocations FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_get_shmem_allocations() FROM PUBLIC;
+CREATE VIEW pg_backend_memory_contexts AS+ SELECT * FROM pg_get_backend_memory_contexts();+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.cindex abda22fa57..b0732ec560 100644--- a/src/backend/utils/mmgr/mcxt.c+++ b/src/backend/utils/mmgr/mcxt.c@@ -21,12 +21,13 @@
#include "postgres.h"
+#include "funcapi.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/builtins.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
-
/*****************************************************************************
* GLOBAL MEMORY *
*****************************************************************************/
@@ -67,6 +68,12 @@ static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
#define AssertNotInCriticalSection(context) \
Assert(CritSectionCount == 0 || (context)->allowInCritSection)
+/* ----------+ * The max bytes for showing identifiers of MemoryContext.+ * ----------+ */+#define MEMORY_CONTEXT_IDENT_SIZE 1024+
/*****************************************************************************
* EXPORTED ROUTINES *
*****************************************************************************/
@@ -1220,3 +1227,126 @@ pchomp(const char *in)
n--;
return pnstrdup(in, n);
}
++/*+ * PutMemoryContextsStatsTupleStore+ * One recursion level for pg_get_backend_memory_contexts.+ */+static void+PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,+ TupleDesc tupdesc, MemoryContext context,+ MemoryContext parent, int level)+{+#define PG_GET_MEMORY_CONTEXTS_COLS 9+ Datum values[PG_GET_MEMORY_CONTEXTS_COLS];+ bool nulls[PG_GET_MEMORY_CONTEXTS_COLS];+ MemoryContextCounters stat;+ MemoryContext child;+ const char *name = context->name;+ const char *ident = context->ident;++ if (context == NULL)+ return;++ /*+ * To be consistent with logging output, we label dynahash contexts+ * with just the hash table name as with MemoryContextStatsPrint().+ */+ if (ident && strcmp(name, "dynahash") == 0)+ {+ name = ident;+ ident = NULL;+ }++ /* Examine the context itself */+ memset(&stat, 0, sizeof(stat));+ (*context->methods->stats) (context, NULL, (void *) &level, &stat);++ memset(values, 0, sizeof(values));+ memset(nulls, 0, sizeof(nulls));++ values[0] = CStringGetTextDatum(name);++ if (ident)+ {+ int idlen = strlen(ident);+ char clipped_ident[MEMORY_CONTEXT_IDENT_SIZE];++ /*+ * Some identifiers such as SQL query string can be very long,+ * truncate oversize identifiers.+ */+ if (idlen >= MEMORY_CONTEXT_IDENT_SIZE)+ idlen = pg_mbcliplen(ident, idlen, MEMORY_CONTEXT_IDENT_SIZE - 1);++ memcpy(clipped_ident, ident, idlen);+ clipped_ident[idlen] = '\0';+ values[1] = CStringGetTextDatum(clipped_ident);+ }+ else+ nulls[1] = true;++ if (parent == NULL)+ nulls[2] = true;+ else+ values[2] = CStringGetTextDatum(parent->name);+ values[3] = Int32GetDatum(level);+ values[4] = Int64GetDatum(stat.totalspace);+ values[5] = Int64GetDatum(stat.nblocks);+ values[6] = Int64GetDatum(stat.freespace);+ values[7] = Int64GetDatum(stat.freechunks);+ values[8] = Int64GetDatum(stat.totalspace - stat.freespace);+ tuplestore_putvalues(tupstore, tupdesc, values, nulls);++ for (child = context->firstchild; child != NULL; child = child->nextchild)+ {+ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,+ child, context, level + 1);+ }+}++/*+ * pg_get_backend_memory_contexts+ * SQL SRF showing backend memory context.+ */+Datum+pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)+{+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;+ TupleDesc tupdesc;+ Tuplestorestate *tupstore;+ MemoryContext per_query_ctx;+ MemoryContext oldcontext;++ /* check to see if caller supports us returning a tuplestore */+ if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("set-valued function called in context that cannot accept a set")));+ if (!(rsinfo->allowedModes & SFRM_Materialize))+ ereport(ERROR,+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),+ errmsg("materialize mode required, but it is not allowed in this context")));++ /* Build a tuple descriptor for our result type */+ if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE)+ elog(ERROR, "return type must be a row type");++ per_query_ctx = rsinfo->econtext->ecxt_per_query_memory;+ oldcontext = MemoryContextSwitchTo(per_query_ctx);++ tupstore = tuplestore_begin_heap(true, false, work_mem);+ rsinfo->returnMode = SFRM_Materialize;+ rsinfo->setResult = tupstore;+ rsinfo->setDesc = tupdesc;++ MemoryContextSwitchTo(oldcontext);++ PutMemoryContextsStatsTupleStore(tupstore, tupdesc,+ TopMemoryContext, NULL, 0);++ /* clean up and return the tuplestore */+ tuplestore_donestoring(tupstore);++ return (Datum) 0;+}diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.datindex 38295aca48..a48982edad 100644--- a/src/include/catalog/pg_proc.dat+++ b/src/include/catalog/pg_proc.dat@@ -7797,6 +7797,15 @@
proargnames => '{name,off,size,allocated_size}',
prosrc => 'pg_get_shmem_allocations' },
+# memory context of local backend+{ oid => '2282', descr => 'statistics: information about all memory contexts of local backend',+ proname => 'pg_get_backend_memory_contexts', prorows => '100', proretset => 't',+ provolatile => 'v', proparallel => 'r', prorettype => 'record', proargtypes => '',+ proallargtypes => '{text,text,text,int4,int8,int8,int8,int8,int8}',+ proargmodes => '{o,o,o,o,o,o,o,o,o}',+ proargnames => '{name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes}',+ prosrc => 'pg_get_backend_memory_contexts' },+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.outindex b813e32215..30ac197af7 100644--- a/src/test/regress/expected/rules.out+++ b/src/test/regress/expected/rules.out@@ -1324,6 +1324,16 @@ pg_available_extensions| SELECT e.name,
e.comment
FROM (pg_available_extensions() e(name, default_version, comment)
LEFT JOIN pg_extension x ON ((e.name = x.extname)));
+pg_backend_memory_contexts| SELECT pg_get_backend_memory_contexts.name,+ pg_get_backend_memory_contexts.ident,+ pg_get_backend_memory_contexts.parent,+ pg_get_backend_memory_contexts.level,+ pg_get_backend_memory_contexts.total_bytes,+ pg_get_backend_memory_contexts.total_nblocks,+ pg_get_backend_memory_contexts.free_bytes,+ pg_get_backend_memory_contexts.free_chunks,+ pg_get_backend_memory_contexts.used_bytes+ FROM pg_get_backend_memory_contexts() pg_get_backend_memory_contexts(name, ident, parent, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes);
pg_config| SELECT pg_config.name,
pg_config.setting
FROM pg_config() pg_config(name, setting);
--
2.18.1
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: torikoshia@oss.nttdata.com, masao.fujii@oss.nttdata.com
Subject: Re: Creating a function for exposing memory usage of backend process
In-Reply-To: <cd0e961fd42e5708fdea70f7420bf214@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox